I'm voting for solution (1). Otherwise, what would be the point of having a "run" dependency?

Also, I think the current manual is somewhat misleading on this situation. Specifically,
http://spack.readthedocs.io/en/latest/packaging_guide.html#dependency-types states that:

“run”: the project is used by the project at runtime. The package will be added to PATH and PYTHONPATH

Strongly agree with (1), I asked that question some time ago, but it didn't gain any traction (and I think there even was a reason given for why not, but I don't remember and probably didn't understand it completely). Being able to use spack load <package> instead of spack load -r <package> is imo more natural

Although they are listed as type='run' dependencies in the package, this information isn't used anywhere during modulefile generation (as far as I know).

That's correct. Branches of different types (build, run, link, tests) were added after module support, and module generation has never been updated with respect to that. So I think you're right: it's time we add something that makes good use of that information when generating modules. I'll try to come up with a proposal (and push in the meanwhile for #7098 to go forward).

@healther @matthiasdiener Autoloading comes with its own drawbacks. This is just to say that autoloading might be the default, if people agree with that, but certainly not the only choice. I think module generation should remain highly configurable as sites that use modules seem to be very opinionated on how they want to deploy them.

@alalazo: As we introduced spack view that solved my problem I haven't looked to hard into modules anymore. Take my comments more as a "I don't understand why someone wants it that way" rather than a "wanting it that way is stupid" :)

As an aside, spack load -r does not seem to be documented at all. https://spack.readthedocs.io/en/latest/command_index.html#spack-load only mentions an -h|--help option, which actually only forwards this option to the module command.

yeah, we could add a help page, but if someone would be willing to put in the work to actually do a clean implementation that would be better. I'll probably have some time on Friday, so if no one has shouted here I'll do the documentation :)

For what it's worth, here is the Lmod documentation for handling dependencies.

It seems like our default Lmod configuration should minimally include prereq(...) statements for the run-time dependencies, so that Lmod generates useful error messages instead of allowing the user to experience run-time failures.

There are two other choices: load() and always_load(). Modules loaded with load() are unloaded if/when the dependee is unloaded. Modules loaded with always_load() are not unloaded if/when the dependee is unloaded.

In an ideal world, I think that what I would want is to unload the dependent if it wasn't loaded explicitly and is no longer used by other modulefiles.

In an ideal world, I think that what I would want is to unload the dependent if it wasn't loaded explicitly and is no longer used by other modulefiles.

Agreed. Of the options listed, I think depends_on() is the closest to the behavior I would expect.

[edit, typo]

I made a quick hack to the modulefile.lua template file so that it uses depends_on instead of load.

diff --git a/templates/modules/modulefile.lua b/templates/modules/modulefile.lua
index cf37595..9b19db5 100644
--- a/templates/modules/modulefile.lua
+++ b/templates/modules/modulefile.lua
@@ -61,12 +61,7 @@ setenv("LMOD_{{ name|upper() }}_VERSION", "{{ version_part }}")

 {% block autoloads %}
 {% for module in autoload %}
-if not isloaded("{{ module }}") then
-{% if verbose %}
-    LmodMessage("Autoloading {{ module }}")
-{% endif %}
-    load("{{ module }}")
-end
+depends_on("{{ module }}")
 {% endfor %}
 {% endblock %}

It seems to work quite nicely. Both gcc and curl depend on zlib and you can see in the example below that it's not unloaded until it's safe.

[hartzell@ip-10-96-1-107 spack]$ module purge
[hartzell@ip-10-96-1-107 spack]$ module load gcc
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa



[hartzell@ip-10-96-1-107 spack]$ module load emacs
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa     4) nettle/3.3-gm37wru    7) libxml2/2.9.8-hgfwbir  10) gettext/0.19.8.1-vf4khxi  13) emacs/26.1-nyfkrps
  2) ncurses/6.1-b3yjgmv   5) zlib/1.2.11-56rgaos   8) bzip2/1.0.6-cwlahss    11) gnutls/3.5.13-4oa5gq7
  3) gmp/6.1.2-cd47nos     6) xz/5.2.4-xz3h7mc      9) tar/1.30-ah7apno       12) pcre/8.42-idvaazk



[hartzell@ip-10-96-1-107 spack]$ module load curl
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa     4) nettle/3.3-gm37wru    7) libxml2/2.9.8-hgfwbir  10) gettext/0.19.8.1-vf4khxi  13) emacs/26.1-nyfkrps
  2) ncurses/6.1-b3yjgmv   5) zlib/1.2.11-56rgaos   8) bzip2/1.0.6-cwlahss    11) gnutls/3.5.13-4oa5gq7     14) openssl/1.0.2n-66jppsd
  3) gmp/6.1.2-cd47nos     6) xz/5.2.4-xz3h7mc      9) tar/1.30-ah7apno       12) pcre/8.42-idvaazk         15) curl/7.60.0-mnkg6zw



[hartzell@ip-10-96-1-107 spack]$ module load jq
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa     4) nettle/3.3-gm37wru    7) libxml2/2.9.8-hgfwbir  10) gettext/0.19.8.1-vf4khxi  13) emacs/26.1-nyfkrps      16) oniguruma/6.1.3-mo2fout
  2) ncurses/6.1-b3yjgmv   5) zlib/1.2.11-56rgaos   8) bzip2/1.0.6-cwlahss    11) gnutls/3.5.13-4oa5gq7     14) openssl/1.0.2n-66jppsd  17) jq/1.5-bic4vte
  3) gmp/6.1.2-cd47nos     6) xz/5.2.4-xz3h7mc      9) tar/1.30-ah7apno       12) pcre/8.42-idvaazk         15) curl/7.60.0-mnkg6zw



[hartzell@ip-10-96-1-107 spack]$ module unload emacs
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa   2) zlib/1.2.11-56rgaos   3) openssl/1.0.2n-66jppsd   4) curl/7.60.0-mnkg6zw   5) oniguruma/6.1.3-mo2fout   6) jq/1.5-bic4vte



[hartzell@ip-10-96-1-107 spack]$ module unload curl
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa   2) oniguruma/6.1.3-mo2fout   3) jq/1.5-bic4vte



[hartzell@ip-10-96-1-107 spack]$ module unload jq
[hartzell@ip-10-96-1-107 spack]$ module list

Currently Loaded Modules:
  1) gcc/5.5.0-26dw7wa



[hartzell@ip-10-96-1-107 spack]$

@alalazo

• What do you think of switching the default template from using load() to using depends_on()?
• Is there a way to have a "local" copy of the template (like we do for the various YAML files) or would I just override overwrite it?

Is there a way to have a "local" copy of the template (like we do for the various YAML files) or would I just override it?

Yes, @alalazo added logic so that you can override it. See http://spack.readthedocs.io/en/latest/tutorial_modules.html#working-with-templates

@alalazo added logic

Brilliant!

@adamjstewart Pleaser review #2982 for a previous discussion of this issue.

@citibeth I think a lot has changed since #2982. Now that depends_on has a type='run' argument, we now have a way of directly knowing which packages need to be recursively loaded.

I think #2982 was the right idea and agree with @adamjstewart's great description here that we should autoload run dependencies by default. It looks like #2982 was aimed at implementing a lot of this stuff in spack, but really as @adamjstewart says, the module systems should handle this based on the dependency relationships Spack already has.

To be honest, I assumed this was easy to configure and that we had this right in the modules implementation. Spack was designed with compiled packages in mind, so we go to great lengths to get RPATH right. So, sorry for not looking more in-depth at this. Loading what is needed for packages to run should be the default and shouldn't require configuration.

Also, FYI: @becker33 and I visited @alalazo at EPFL week before last and we set up a project to improve modules support. This issue is in it. More here.

@citibeth: lmod is reference counted (see docs above). I assume that tcl modules do something similar but I am not sure. Both store state in the environment.

It's going to happen :)

@hartzell

What do you think of switching the default template from using load() to using depends_on()?

I am looking into that right now. It seems fine to me at a first glance. There's a known caveat:

When gcc is unloaded it removes the path in MODULEPATH to the gcc dependent modules which means that X will be unloaded and marked as inactive. The way that a module is unloaded is that the contents of the module is evaluated and most action requested are reversed. So load statements cause a module to unload and a depends_on() function is told to forgo() the modules. The isloaded() is not reversed. But as you can see since the gcc modulefile is not loaded the if statement then clause is not evaluated. This means that openblas will still be loaded.

but I don't think we'll ever hit that (famous last words). I need to check since which version of LMod is depends_on supported, but in the worst case we can let a user select which idiom to use for autoloading lmod modules in modules.yaml. What do you think?

@alalazo -- Works for me. I don't think [hey, I'm famous too... :)] that caveat will be a problem for me in my current use-case (aka "job"). Others (@peetsv) might be able to speak to the old use-case (aka "previous gig") but I don't think it'll be a problem for them either.

Are there any news here?
I agree with the argument that loading modules for every dependency might kill performance in many applications especially HPC, as @citibeth just mentioned. But I also believe this is mostly problematic with extensions (like Py packages) which are loaded by the dozens. Eventually in the case of pure dependencies we are just fine recursively loading modules.

Maybe it wouldn't be such a bad idea to only optimize the case of extensions (to which many sites don't like creating modules), so their environment setup would always be "merged" into the 'requiring' software environment.

Hi! Any update regarding this issue?

Hi @citibeth, thanks for your reply, but I was referring to the "autoload" of dependencies of type "run".
In particular, I tried adding the "run" type to regcm as requested also by issue #10163, but spack didn't add the /bin/ directory of the dependency to the module generated for regcm, see discussion with Adam in PR #10863.

Yes, spack environments are a great tool for single users, but for sys admins who want to deploy a set of modules that "Just Work", we still have a long ways to go with autoloading run deps.

As an aside to the module discussion:

Let me point out that the -r flag in spack load -r has no effect whatsoever, it's an unused flag, probably an oversight. See #25731. The current default of spack load x is to load x + all dependencies, including build deps of build deps of build deps, which doesn't make sense.

And then if #25732 gets merged too, spack load x will do exactly what this thread wants module load x to do, namely load x + runtime deps. This would then be similar to spack env activate x, which only loads root specs of env x + runtime deps.