Couldn't resist making a meme for this PR...

In all seriousness, I support this PR, and think it's the right direction to go if we want to continue to make Spack easy to set up and use. My only concern is that Environment Modules/Lmod are massive projects, and we're attempting to roll our own module management system here. How much of Lmod's features do we plan on supporting? How large of a codebase will this become? Using a wrapper around module load introduces complexity on its own (not sure if we even support Env Mod 4 yet), so maybe this is actually simpler than it used to be. Just want to make sure we aren't digging ourselves deeper here.

@adamjstewart I think we get most of this feature "for free" from other things we have to do. We already need to know the environment changes necessary to load a package (because we generate modules) and we need to know how to reverse those changes (because we deactivate environments). So far, this feature is just a thin shell script shim layer. I do think it makes sense to add tracking for which hashes are loaded, but that's also pretty trivial, and can be hooked up to the find command.

The previous state of affairs:

Will there be something analogous to the module systems ability to filter things from the modulefiles? E.g.

modules:
  lmod:
    all:
      filter:
        environment_blacklist: 'LD_LIBRARY_PATH'
    bowtie2:
      environment:
        LD_LIBRARY_PATH: '$prefix/lib'

Some of us (ok, at least I...) consider setting LD_LIBRARY_PATH and etc. in the modulefiles to be an antipattern, one that throws away all of the work Spack does to get the RPATH bits correct.

This was pretty widely discussed back in #3955.

I recognize the utility of setting those variables if you're supporting folks who are linking against libraries in the Spack tree. I, on the other hand, am simply trying to use the things I've built with Spack with minimal side effects. #3955 documents LD_LIBRARY_PATH breaking yum, we ran into other cases on the linux boxes and I've since tripped over issues on my Mac that were fixed by adding DYLD_LIBRARY_PATH to the blacklist.

I've been running with this for a while now, without the exception for python, and haven't been having any trouble.

modules:
  lmod:
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH', 'LD_LIBRARY_PATH', 'DYLD_LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'

Sorry, no GIF....

Echo'ing a bit what I hear @adamjstewart asking, I'm worried about Spack trying to be do too many things.

The only feature that I've seen for Spack load that it's trivial w/ Lmod modules is the ability to cook up an arbitrary spec and have load the appropriate module. In the few cases where my user community needed that feature I've use the module systems ability to add suffixes to provide simple names that my users can figure out w/out documentation, e.g. ...-without-X for the non-X variant of an application. I've never been eager to try to explain Spack's spec language to my end-user communities.

Beyond that it seems that it's duplicating existing work and building a monolithic system.

I suppose what I'm asking is: what is the end goal for this set of features?

• Will I be able to use it to also set up things that aren't part of Spack?
• Will is support blacklisting and etc?
• Can we give things easy-to-understand names, synonyms?
• Will it support hierarchical modules that make foot-shooting difficult?
  • If it does, will it fix Lmod's/our problems with things that are built with e.g. D and end up in their only branch of the tree?

I suppose what I'm asking is: what is the end goal for this set of features?

* Will I be able to use it to also set up things that aren't part of Spack?

* Will is support blacklisting and etc?

* Can we give things easy-to-understand names, synonyms?

* Will it support hierarchical modules that make foot-shooting difficult?
  
  * If it does, will it fix Lmod's/our problems with things that are built with e.g. D and end up in their only branch of the tree?

I do not think we should enable this for non-spack software. We will still generate modules for Spack-installed software, and those modules will be interoperable with the greater module ecosystem.

In this PR I've refactored the prefix_inspections from lib/spack/spack/environment.py. The plan is to merge those with the module configuration prefix inspections and give them a shared configuration format at some later date.

On your last two points, the answer to both is environments. Spack allows environments with pithy names, and an environment with concretization: together manages coherency to avoid all foot-shooting incidents. The effect of spack load foo in this PR is exactly the same as spack env activate foo-env, where foo-env is an environment containing only the package foo.

close-cycling to restart the tests, as they didn't start properly on the last push

The only feature that I've seen for Spack load that it's trivial w/ Lmod modules is the ability to cook up an arbitrary spec and have load the appropriate module.

@hartzell: not sure if it's clear from the PR, but the only thing spack load and spack unload ever did was load/unload a single spec -- using the module system. The only thing this PR does is make it so that we don't need modules to do that.

We're definitely not getting rid of modules; all those customizations will still be possible by generating modules, and we'll still do that. Users can still use modules through the module command, not through spack load. Do your users actually use spack load? If not, this won't affect them.

The overarching goal here is to give users some easy ways to use Spack that a) don't require setting up an entire module system, and b) work the same on your laptop, HPC machine, etc. spack load is for loading a one-off and spack environments provide more consistency (via concretization) than modules can actually provide. Lmod's a hierarchy and it keeps you safe for MPI and your compiler -- dependency resolution for environments handles the whole DAG.

Does this and @becker33's comment alleviate your concerns?

[edit: rainbox -> rainbows]

@tgamblin --

Does this and @becker33's comment alleviate your concerns?

The main alleviant [SIC?] is the trust that I have that all y'all will
build something that works.

@hartzell: not sure if it's clear from the PR, but the only thing
spack load and spack unload ever did was load/unload a single spec --
using the module system. The only thing this PR does is make it so
that we don't need modules to do that.

Well..... There are a bunch of Issues and PR's that talk about how
loading a single modulefile isn't really very useful and we have to
load their dependencies (link/run, run, what-have-you) in order to do
anything useful. Then there's spack load -r.

Seems like a slippery slope; writing and supporting more code that
works just up until there's a sudden phase change and users hear
"Well, you need to use modules/environments/activate/*".

[...]
The overarching goal here is to give users some easy ways to use Spack
that a) don't require setting up an entire module system, and b) work
the same on your laptop, HPC machine, etc. [...]

Sure, but a hypothetical/mythical

spack install emacs
spack install lmod
. $(spack setup-lmod --with-unicorn --double-rainbows --but-seriously)
module load emacs

would (at the cost of requiring lmod in addition to whatever else
you've built) provide a more continuous path towards a fluidly
functioning system (e.g. one that doesn't need a chain activations
in order to get flake8 to run).

I'll bet a beer if/when we ever get to meet that eventually spack load
will have the bulk of the functionality of Lmod.

Rather than make spack load work without modules, why not use
modules to make spack work w/out spack load? Yikes, keep reading!

But, the more I argue, the more I put the lie to my opening statement.

Second guessing and arm chair quarterbacking...

There's more than one way to do things and every set of choices
comes complete with its own special corner for painting-into. Your
engineering intuition, use cases, users, available resources, and
priorities are yours and mine are mine. I live with (and perhaps even
have learned to like) the squeaks, groans, and itches that my
approaches generate, but it is known that it all still squeaks,
groans, and itches....

You've built something great and I'm thankful to have it in my
toolbox.

clarification on this:

I figure we can save modifying setup-env.sh for a separate PR -- do you agree?

I am talking about the part that looks for an lmod/modules installation. I know you already modified load and unload.

@tgamblin yes I agree. I will look over the documentation either tonight or tomorrow.

This is going to require a bunch or rewriting documentation. I'll get started on it tomorrow.

Just tested this branch (doing more homework for #14348). If a package is renamed or deleted (but is installed), it will no longer 'spack load'.

@aweits: the installed-package issue is an issue with the module-based spack load, so we'll fix that one in a separate PR.

This change completely alters our workflow. We liked how "spack load -r packagename" loaded the requested module and all its dependencies. we also liked that it abstracted over environment-modules vs lmod.

spack module lmod loads margo does not work for us on summit, telling us "no module available", even though "spack load margo" gives no such error (and "spack load margo" does not load dependencies).

Suppose we were to ignore the lmod modules and let spack manage what packages are loaded on its own. Once we have done a spack load -r margo (or similar), how can we list it and any other packages it may have loaded (e.g., the equivalent of what we would have done with module list if using lmod)? For example, if someone wants to be able to switch between versions with different variant flags, or different cflags?

For some context, we need to rapidly figure out how to fix our documented workflow for a tutorial. One solution would be to go back to a tagged spack release, but then we lose updates to packages that are only present in origin/develop because package descriptions are embedded in the spack repo.

We liked how "spack load -r packagename" loaded the requested module and all its dependencies. we also liked that it abstracted over environment-modules vs lmod.

This PR actually doesn’t change how modules are generated, and it doesn’t tie anything to lmod - it just makes spack load do what modules would’ve done, without going through modules. If you want tcl modules, what about:

spack module tcl loads margo

Or just load the package using module load based on what’s in module avail? All the same modules should be there.

Once we have done a spack load -r margo (or similar), how can we list it and any other packages it may have loaded (e.g., the equivalent of what we would have done with module list if using lmod)?

Try:

spack find --loaded

That will give you the list of loaded packages. You can unload them with spack unload, or all of them with spack unload -a.

The other way you could do this is to build your package in an environment, and spack env activate and spack env deactivate the environment.

I’m a bit confused about the use of lmod here, as spack load was always searching tcl modules. I guess it was going through Lmod because that’s what was installed on summit, but what it actually loaded was the tcl module file (which lmod understands) — so also try my suggestion above to use spack module tcl loads, as that may require the least number of changes.

The spack find --loaded option is probably helpful in the short term; we'll try that out.

spack module tcl loads margo is a kind of handful for users to remember in a tutorial context, especially if what it really translates to in order to enact the load is something even more verbose like source <(spack module tcl loads --dependencies margo)

We have a few meta-problems to circle back on later, but I'll comment on quickly here:

The spack documentation is clear that it shouldn't be necessary to load dependencies (and indeed we can see that rpath has correct paths where needed), but all of our packages also use pkgconfig, and right now nothing is putting the .pc dependencies in place unless we instruct spack to load dependencies.

More generally when we run into these kinds of speed bumps, we can usually find a quick solution (like the find --loaded) above, but the fluidity of the syntax / best practice is a challenge from our perspective when we need to track origin/develop to get up to date package versions. In a perfect world I would like to lock onto a stable released version of spack itself while still having access to updated packages so that it's easier to pick up new versions of packages without risking breakage in our overall workflow at an inopportune moment.

As it stands we've had some bad luck with the timing of changes to spack behavior. In this case for example, its not that we care about lmod, its just that we expected to be able to module list to see what's going on, and it stopped working one day without it being immediately obvious what the replacement was.

With the previous behavior one could use module save and module restore to keep track of the exact environment (which is a mix of spack and non-spack modules). The new approach makes it impossible. Does anyone has workarounds for that?

The new approach makes it impossible.

Is it impossible, or impossible with spack load? You can still do all that by module load-ing what you want.

You are right, I meant impossible with spack load. I was trying to avoid spack module lmod loads thingie which seems to behave differently (Error: the key "core_compilers" must be set in modules.yaml). I guess I'll need to fix that. Thanks.

Here's a gist that demonstrates an alternative approach, if you're going to go the lmod route.

It doesn't expose the user to the spack command (although it uses it to discover the installed location of the lmod package). Users liked it because they were already familiar with lmod and it gave me a clean way to use modulefiles to set up other non-spack (sigh) software.

In my environment I had a plethora of spack trees, built on different dates w/ different sets of packages for different use cases. The name of the "default" release lives in a file named current, users can override it by setting APPS_DIR before sourcing the script. It assumes that the core compiler is gcc (overridable). There are a bunch of other knobs for adventurous users.

Here's some of the commentary from the script:

# The typical user's .bash_profile would include:
#
# APPS_MODULES="emacs git htop the-silver-searcher"   # and etc....
# source /moose/apps/init.sh
#
# Even setting APPS_MODULES is optional if user is ok with default list...
#
# That will:
#
# 1. Fetch the name of the "current" spack tree from the Well Known
#    File if the user hasn't explicitly provided one.
# 2. Use the spack executable in that tree to ask for the location of
#    the Lmod directory in that tree.
# 3. Load the Lmod bash initialization file from within that Lmod dir.
# 4. Tell Lmod to use the Core dir that spack built.
# 5. Tell Lmod to use our directory of handcrafted module files.
# 6. Load the core compiler module, which makes the bulk of the
#    modulefiles accessible.
# 7. Load the user's modulefiles.
#
# There are a few things that users can customize:
# Common:
#   APPS_MODULES - your list of modules to load
#      e.g. APPS_MODULES="emacs git htop etc etc etc..."
# Less common:
#   APPS_DIR - path to top of a Spack apps tree, e.g. a team specific tree
#              (must include lmod)
#   APPS_SPECIAL_MODULES_DIR - path to dir of snowflake modulefiles
# Uncommon:
#   APPS_CORE_COMPILER - Compiler choice for lmod hierarch. scheme
#