Modules: Always autoload run dependencies #8639
Replies: 31 comments 1 reply
-
|
I'm voting for solution (1). Otherwise, what would be the point of having a "run" dependency? Also, I think the current manual is somewhat misleading on this situation. Specifically,
|
Beta Was this translation helpful? Give feedback.
-
|
Strongly agree with (1), I asked that question some time ago, but it didn't gain any traction (and I think there even was a reason given for why not, but I don't remember and probably didn't understand it completely). Being able to use |
Beta Was this translation helpful? Give feedback.
-
That's correct. Branches of different types ( |
Beta Was this translation helpful? Give feedback.
-
|
@healther @matthiasdiener Autoloading comes with its own drawbacks. This is just to say that autoloading might be the default, if people agree with that, but certainly not the only choice. I think module generation should remain highly configurable as sites that use modules seem to be very opinionated on how they want to deploy them. |
Beta Was this translation helpful? Give feedback.
-
|
@alalazo: As we introduced |
Beta Was this translation helpful? Give feedback.
-
|
As an aside, |
Beta Was this translation helpful? Give feedback.
-
|
yeah, we could add a help page, but if someone would be willing to put in the work to actually do a clean implementation that would be better. I'll probably have some time on Friday, so if no one has shouted here I'll do the documentation :) |
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth, here is the Lmod documentation for handling dependencies. It seems like our default Lmod configuration should minimally include There are two other choices: In an ideal world, I think that what I would want is to unload the dependent if it wasn't loaded explicitly and is no longer used by other modulefiles. |
Beta Was this translation helpful? Give feedback.
-
Agreed. Of the options listed, I think |
Beta Was this translation helpful? Give feedback.
-
|
[edit, typo] I made a quick hack to the diff --git a/templates/modules/modulefile.lua b/templates/modules/modulefile.lua
index cf37595..9b19db5 100644
--- a/templates/modules/modulefile.lua
+++ b/templates/modules/modulefile.lua
@@ -61,12 +61,7 @@ setenv("LMOD_{{ name|upper() }}_VERSION", "{{ version_part }}")
{% block autoloads %}
{% for module in autoload %}
-if not isloaded("{{ module }}") then
-{% if verbose %}
- LmodMessage("Autoloading {{ module }}")
-{% endif %}
- load("{{ module }}")
-end
+depends_on("{{ module }}")
{% endfor %}
{% endblock %}It seems to work quite nicely. Both
|
Beta Was this translation helpful? Give feedback.
-
Yes, @alalazo added logic so that you can override it. See http://spack.readthedocs.io/en/latest/tutorial_modules.html#working-with-templates |
Beta Was this translation helpful? Give feedback.
-
Brilliant! |
Beta Was this translation helpful? Give feedback.
-
|
@adamjstewart Pleaser review #2982 for a previous discussion of this issue. |
Beta Was this translation helpful? Give feedback.
-
|
@citibeth I think a lot has changed since #2982. Now that |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reading and digesting it!
…On Wed, Jul 18, 2018 at 12:23 PM, Adam J. Stewart ***@***.***> wrote:
@citibeth <https://github.com/citibeth> I think a lot has changed since
#2982 <#2982>. Now that depends_on
has a type='run' argument, we now have a way of directly knowing which
packages need to be recursively loaded.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/spack/spack/issues/8639#issuecomment-405991495>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cdwucgFgpSixRLcvXA-dV-Q7tMiTuks5uH2ESgaJpZM4VB3kz>
.
|
Beta Was this translation helpful? Give feedback.
-
|
I think #2982 was the right idea and agree with @adamjstewart's great description here that we should autoload run dependencies by default. It looks like #2982 was aimed at implementing a lot of this stuff in spack, but really as @adamjstewart says, the module systems should handle this based on the dependency relationships Spack already has. To be honest, I assumed this was easy to configure and that we had this right in the modules implementation. Spack was designed with compiled packages in mind, so we go to great lengths to get |
Beta Was this translation helpful? Give feedback.
-
|
Also, FYI: @becker33 and I visited @alalazo at EPFL week before last and we set up a project to improve modules support. This issue is in it. More here. |
Beta Was this translation helpful? Give feedback.
-
|
Lower-level modules are used many times in a typical DAG. How do we ensure
they are not loaded many times with recursive loading?
|
Beta Was this translation helpful? Give feedback.
-
|
@citibeth: |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure either, maybe @alalazo knows?
In any case... what I REALLY want is Spack Environments to generate a
single module that loads the entire environment. That would be efficient
especially on supercomputers with slow FS. And it would eliminate all
those hashes from anything user-facing.
But first I'm waiting for Spack Environments to get merged....??????
.. Elizabeth
…On Wed, Jul 18, 2018 at 1:27 PM, Todd Gamblin ***@***.***> wrote:
@citibeth <https://github.com/citibeth>: lmod is reference counted (see
docs above). I assume that tcl modules do something similar but I am not
sure. Both store state in the environment.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/spack/spack/issues/8639#issuecomment-406011072>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd-XMF3zi-dU5Z0KxizCs-46uecQIks5uH3AJgaJpZM4VB3kz>
.
|
Beta Was this translation helpful? Give feedback.
-
|
It's going to happen :) |
Beta Was this translation helpful? Give feedback.
-
I am looking into that right now. It seems fine to me at a first glance. There's a known caveat:
but I don't think we'll ever hit that (famous last words). I need to check since which version of |
Beta Was this translation helpful? Give feedback.
-
|
@alalazo -- Works for me. I don't think [hey, I'm famous too... :)] that caveat will be a problem for me in my current use-case (aka "job"). Others (@peetsv) might be able to speak to the old use-case (aka "previous gig") but I don't think it'll be a problem for them either. |
Beta Was this translation helpful? Give feedback.
-
|
Are there any news here? Maybe it wouldn't be such a bad idea to only optimize the case of extensions (to which many sites don't like creating modules), so their environment setup would always be "merged" into the 'requiring' software environment. |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Any update regarding this issue? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, spack environments are now merged. The command ‘spack env loads -r’
will generate a shell script that will load the complete list of modules
you need including dependencies.
…On Wed, Mar 13, 2019 at 07:08 Alberto Chiusole ***@***.***> wrote:
Hi! Any update regarding this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/spack/spack/issues/8639#issuecomment-472378783>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd0HlEP8SAQ1Ki4OMWJROx0WBXwWaks5vWNxBgaJpZM4VB3kz>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @citibeth, thanks for your reply, but I was referring to the "autoload" of dependencies of type "run". |
Beta Was this translation helpful? Give feedback.
-
|
Yes, spack environments are a great tool for single users, but for sys admins who want to deploy a set of modules that "Just Work", we still have a long ways to go with autoloading run deps. |
Beta Was this translation helpful? Give feedback.
-
|
Spack environments are exactly the tool we have available to describe a set
of modules that will “just work”. As Todd has pointed out, manually loaded
modules are fundamentally broken. No amount of auto loading will fix this.
…On Wed, Mar 13, 2019 at 15:17 Adam J. Stewart ***@***.***> wrote:
Yes, spack environments are a great tool for single users, but for sys
admins who want to deploy a set of modules that "Just Work", we still have
a long ways to go with autoloading run deps.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/spack/spack/issues/8639#issuecomment-472567110>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd2RtM_p3FN1eQaSIZWeuj9HII6-fks5vWU7AgaJpZM4VB3kz>
.
|
Beta Was this translation helpful? Give feedback.
-
|
As an aside to the module discussion: Let me point out that the And then if #25732 gets merged too, |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
One of the tenets of Spack is the use of RPATH. It allows packages to find a complicated set of dependencies and prevents users from loading incompatible dependencies. Unfortunately, RPATH doesn't extend to non-compiled languages. Python, Perl, R, and Java all require their dependencies to be found via
PYTHONPATH,PERL5LIB,R_LIBS, andCLASSPATH, respectively. As a package manager, Spack is great at managing dependencies. This should extend to its auto-generated module files.Expected Result
If I need to use a software package, I should be able to
module loadthat package and go.Actual Result
For many packages, I also need to
module loadall of their runtime dependencies. For example, if I want to build our docs, I need to:The situation for
py-matplotlibis even worse. And then there'sr-rminer...Existing Alternatives
There are some ways to get around this issue. Spack already has builtin support for autoloading dependencies: http://spack.readthedocs.io/en/latest/module_file_support.html#autoload-dependencies
However, this requires manual intervention on the part of the Spack admin. It also needs to be done for every non-compiled programming language. There are also many compiled programs that pick up runtime dependencies via
PATH. Although they are listed astype='run'dependencies in the package, this information isn't used anywhere during modulefile generation (as far as I know).One can also
spack activateextendable packages like Python and R. Again, this only works for extendable packages; there are many non-extendable packages that suffer from the same problem.spack viewis another alternative, but not always possible when multiple configurations of the same package are installed.One can
spack load -rto recursively load dependencies, but this will load many unnecessary dependencies when the only thing you actually need are 'run' dependencies.Many packages have taken matters into their own hands. See #8629 for one of dozens of packages that manually put their 'run' dependencies in the
PATH.Proposed Solutions
I see 2 possible solutions to this dilemma. Either one of these should be enabled by default (or even non-negotiable). This will greatly reduce the difficulty that new users face when starting to use Spack.
PATH,PYTHONPATH, etc.Can anyone think of arguments for or against 1 or 2? Is there any reason Spack shouldn't always autoload 'run' dependencies?
@tgamblin @alalazo @hartzell @healther @matthiasdiener
Beta Was this translation helpful? Give feedback.
All reactions