remove activate/deactivate support in favor of environments#29317
remove activate/deactivate support in favor of environments#29317tgamblin merged 1 commit intospack:developfrom
Conversation
haampie
force-pushed
the
remove/activate-deactivate-extensions
branch
from
acf7d31 to
9f20412
Compare
spackbot-app
bot
added
commands
directives
documentation
haampie
force-pushed
the
remove/activate-deactivate-extensions
branch
from
9f20412 to
55a12b6
Compare
|
I didn't even know this existed. Probably not a good sign. I think the legacy behavior of a global spack database is definitely prone to confusion and error. Environments are a much cleaner interface |
@psakievich: Can you elaboration this? activation actually isn't handled in the DB (bookkeeping is slightly worse than that -- it's in |
|
I'm in favor of this. @lee218llnl at LLNL currently uses activation to deploy LLNL python environments, and activation behavior is actually different from linking into an env. Last I talked to @lee218llnl, the thing we need to remove reliance on this feature is to allow for only extendees (i.e. just the python packages) to be linked into the python env, not their non-extension dependencies. @lee218llnl will have to remind me why they want that, but I believe it makes it easier for consumers of the deployed python installation to avoid conflicts. |
|
I think all it would take to implement this is a new linking option for views -- we currently have |
|
@tgamblin I was referring to the difference between spack environments which give a constrained version of all the installed packages that reside in the database vs something like |
I think a better way to say this is that you end up polluting your python package, and it's hard for other things to cleanly depend on it, as the DB (by which I mean the thing in $install_tree_root/.spack-db/index.json) isn't actually affected by this at all. Users at LLNL run into this problem, e.g. with pre-activated versions of packages that are frequently used by the codes. I think as others have said that it makes a lot of sense to instead have environments (that maybe users can load with named modules if they don't want to use spack directly), as that'll keep the python install clean and also allow people to use facility provided python envs. |
|
@tgamblin thanks for keeping me honest! I always learn something with your clarifications. As a non-user of this feature I'm pretty ignorant of the bugs people experience with it. |
Happy to hear that!
I don't understand this. In my view (lol) it would make sense to replace activate with something like this: and then have the option to include just transitive run/link type deps in the environment view. Currently environment view include too much, but putting only some "extendees" in there... seems too little. (Also, it would not hurt to only include transitive link + run type deps in the environment view only by default, right? Given that Also I talked to @becker33 about views, and since environment views have moved away from mutable views (add specs, remove specs from same folder) to a fully new view on every update and an "atomic" symlink change, once we get this change in, a lot of the complexity due to |
I think this is the point, to only add enough to find the python packages' I think |
Actually wrong about this, what environments do right now is the sensible thing, they only include transitive run/link time deps by default (when
Is it too much to ask to just list them all in |
I mean, I think this is the sort of annoying thing that you might submit an issue about in the future. I don't think we should expect the user to know which dependencies are also extendees (esp since that can change on re-concretization). I think this ends up being a fairly simple change in |
Sure, I think it can be done in a separate PR though. |
Sure, but seeing as how without it, this PR will break functionality, that PR will need to happen first. |
|
I think it's easier to just do |
|
Sorry to chime in late. I think Todd accurately captured my use case for Python and its extensions at LLNL. The activate feature was designed largely based on the requirements I had for building Python at LLNL. In particular, we want to provide a Python with a set of site-packages that many of our users require. For specifics, you can see what we have installed at https://hpc.llnl.gov/software/development-environment-software/python/python-site-packages. Spack provides a convenient way for me to build all of this, as I used to manually install everything. However, we don't want users to be bound to this Spack instance and all of the dependent packages that were built to get the desired Python + site-packages. We do not want users to see the dependent packages that were required to build Python when they point their $PATH to wherever the python binary lives. Currently this works as desired, because the only things in the Python bin directory are python itself and all of the bin dirs that were linked upon activation. The problem with a view that includes dependences is that it would include things like HDF5 and we don't want users using that HDF5 instance, as we have separate HDF5 installations that we expect users to use. Does this all make sense? In short, we want a standalone Python with many site-packages that does not otherwise affect a user's expected build environment. I agree with Todd that we don't want to break functionality, even if a future PR will address this. |
|
Can you try #29336? I'm reasonably sure it will be equivalent to what you have currently |
|
Ok, #29336 is in, but I'm reminded that we need to actually deprecate these things for a version before we remove them. So we need a PR for |
|
Ok, so basically merge a deprecation warning right now, and then merge this PR as soon as 0.18 is branched off? |
|
See #29430 |
tgamblin
force-pushed
the
remove/activate-deactivate-extensions
branch
from
55a12b6 to
f44fbe0
Compare
tgamblin
force-pushed
the
remove/activate-deactivate-extensions
branch
from
f44fbe0 to
f5a5101
Compare
) Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons: - Global activation is a really poor idea: - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases. - Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better. Also we write in the docs: ``` However, Spack global activations have two potential drawbacks: #. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, ``spack load openblas`` might be required to make ``py-numpy`` work. #. Global activations "break" a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python --- one with ``[email protected]`` and one with ``[email protected]``. Spack extensions will not support this potential debugging use case. ``` Now that environments are established and views can take over the role of activation non-destructively, we can remove global activation/deactivation.
|
The latest commit from this pull request 0f54a63 leads to following error If I go one commit back, it is fine. Something might be wrong with my setup of course, but it is fresh Spack installation. |
|
Actually, forget it. The issue is there, but it is not coming from this pull request. Sorry for bothering you. |
) Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons: - Global activation is a really poor idea: - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases. - Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better. Also we write in the docs: ``` However, Spack global activations have two potential drawbacks: #. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, ``spack load openblas`` might be required to make ``py-numpy`` work. #. Global activations "break" a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python --- one with ``[email protected]`` and one with ``[email protected]``. Spack extensions will not support this potential debugging use case. ``` Now that environments are established and views can take over the role of activation non-destructively, we can remove global activation/deactivation.
) Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons: - Global activation is a really poor idea: - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases. - Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better. Also we write in the docs: ``` However, Spack global activations have two potential drawbacks: #. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, ``spack load openblas`` might be required to make ``py-numpy`` work. #. Global activations "break" a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python --- one with ``[email protected]`` and one with ``[email protected]``. Spack extensions will not support this potential debugging use case. ``` Now that environments are established and views can take over the role of activation non-destructively, we can remove global activation/deactivation.
The `ignore` parameter was only used for `spack activate/deactivate`, and it isn't used by Spack Environments which have their own handling of file conflicts. We should remove it. Everything that handles `ignore=` was removed in #29317 and included in 0.19, when we removed `spack activate` and `spack deactivate` in favor of environments. So all of these usages removed here were already being ignored by Spack.
This method is vestigial; the only arg we ever used was `ignore=`, and that was eliminated in spack#29317 and spack#35588. The `kwargs` field of the extensions dictionary is actually completely unused now. Add a note for future removal.
This method is vestigial; the only arg we ever used was `ignore=`, and that was eliminated in spack#29317 and spack#35588. The `kwargs` field of the extensions dictionary is actually completely unused now. Add a note for future removal.
This method is vestigial; the only arg we ever used was `ignore=`, and that was eliminated in spack#29317 and spack#35588. The `kwargs` field of the extensions dictionary is actually completely unused now. Add a note for future removal.
5 participants
Environments and environment views have taken over the role of
spack activate/deactivate, and we should deprecate these commands for several reasons:spack view, but better.Also we write in the docs:
Now that environments are established and views can take over the role of activation
non-destructively, we can remove global activation/deactivation.