Modulefiles generated with a template engine by alalazo · Pull Request #3183 · spack/spack

alalazo · 2017-02-19T15:19:41Z

TLDR

This PR is a big refactoring effort on module files (with only little changes on the UI side) that introduces the use of template engines in spack. Iit relies on jinja2 that will be shipped as an external, at least until we solve #1385. I also plan to add a lot more tests on hierarchical module files generation.

Motivation

I started the PR because I needed to extend lmod module file generation to take into account packages that provide more than one service (hello intel-parallel-studio!) and figured out that this part of the code has grown enough in size and complexity to benefit from using a template engine for generation.

I also think that this capability may be reused in many places like:

@adamjstewart spack create command
@citibeth spack loads and spack setup commands
generation of JUnit or CDash conformant files from tests (@tgamblin and @kielfriedt may be interested here)

i.e. basically everywhere we need to generate a file with a clear structure starting from the information in a DAG or in the DB.

Modifications planned (or already done)

Template engine in spack

added an entry template_dirs in config.yaml to list the directories we should search for templates.
hooked jinja2 to spack
copied jinja2 sources into external

Modulefiles

Changes in UI

changed keyword autoload_verbose to just verbose and made it default to False. Its location is at the same level as whitelists and blacklists.

adamjstewart · 2017-02-19T16:00:03Z

This is awesome! Some things to consider:

There are a lot of already open PRs that affect modules. These will have to be rewritten or merged into this PR.
Maybe we could just add jinja2 to lib/spack/external until we get better bootstrapping support.

At ANL's LCRC, we are getting ready to make the switch from our own homegrown SoftEnv package to Lmod. I started playing around with Spack's Lmod support and found a few problems worth fixing. I was going to try to take a stab at some of these, but will obviously wait until this PR is done now. You're welcome to try to solve some of these if you like:

spack module refresh without any specific --module-type should refresh all enabled module types. [low priority]
Users are always asking me how I built a certain package. It would be neat if we could provide the configure or cmake line in the module file somehow. [low priority]
Right now, when I run module avail, I see a giant list of every module I've ever created. NERSC adds some nice formatting that separates things out more. For example, we would want a Core section for things like compilers where we don't care how they were built. We have 2 clusters and 3 MPI fabrics at ANL's LCRC, so we would want to separate those out and possibly only load the MODULEPATH pertaining to that cluster when a user logs in. I'll have to play around a bit more, but I'm not sure if Spack currently handles this case. [medium priority]
spack module refresh needs to work regardless of whether or not I add a package in a branch and switch back to develop, or if a variant gets added/removed in develop. This is currently the biggest problem I've seen. When a user asks me to install something that isn't in Spack yet, I create a new package and install it. I can't tell them "well, I have a package for it, but I can't install it until the next Spack release, sorry". [high priority]
One of the nice features that separates Lmod and Tcl modules is it's ability to control what compilers/MPI libraries are loaded. This involves the use of the family directive, which prevents you from loading multiple compliers/MPI at the same time. It also allows you to "swap" an entire software stack from one compiler to another with module swap gcc intel. I don't think we currently add this family directive. [high priority]

alalazo · 2017-02-19T17:01:13Z

Hey @adamjstewart thanks for the quick feedback! Before I start a by-point reply: feel free to post any problem you encounter with modules (or open an issue an point me to that 😄 ). I'll try to fix what I can here.

About this PR

This is awesome! Some things to consider:

There are a lot of already open PRs that affect modules. These will have to be rewritten or merged into this PR.

I am willing to merge their functionality here (or let the original author of the PR do that if they prefer). For #2902 this is basically done, as templates are designed to be extended. I still need to check what #2345 does (thanks for the ping yesterday). Others?

Maybe we could just add jinja2 to lib/spack/external until we get better bootstrapping support.

That would simplify life! I would like to hear what @tgamblin thinks about it. Currently I used only basic features in templates (i.e. the ones that would have permitted to write a replacement for jinja2 without re-implementing a compiler). If we ship jinja2 as an external we can remove this limitation.

About `lmod` support and issues

spack module refresh without any specific --module-type should refresh all enabled module types.

I agree, will do

Users are always asking me how I built a certain package. It would be neat if we could provide the configure or cmake line in the module file somehow

Also agree.

Right now, when I run module avail, I see a giant list of every module I've ever created. NERSC adds some nice formatting that separates things out more. For example, we would want a Core section for things like compilers where we don't care how they were built. We have 2 clusters and 3 MPI fabrics at ANL's LCRC, so we would want to separate those out and possibly only load the MODULEPATH pertaining to that cluster when a user logs in. I'll have to play around a bit more, but I'm not sure if Spack currently handles this case. [medium priority]

This is currently handled right for a given architecture. You need to specify:

modules:
  lmod:
    core_compilers:
      - '[email protected]'  # Supposing [email protected] is a core compiler

See the tutorial here.

spack module refresh needs to work regardless of whether or not I add a package in a branch and switch back to develop, or if a variant gets added/removed in develop. This is currently the biggest problem I've seen. When a user asks me to install something that isn't in Spack yet, I create a new package and install it. I can't tell them "well, I have a package for it, but I can't install it until the next Spack release, sorry"

This problem is orthogonal to what is done here and that's because even if it is revealed by module files, it is inherently a problem of the database returning to you an inconsistent spec-package pair. I think it has to be solved with high priority, but can be done in another PR.

One of the nice features that separates Lmod and Tcl modules is it's ability to control what compilers/MPI libraries are loaded. This involves the use of the family directive, which prevents you from loading multiple compliers/MPI at the same time. It also allows you to "swap" an entire software stack from one compiler to another with module swap gcc intel. I don't think we currently add this family directive.

Should be already in develop. Actually you can put other services into the mix and have a hierarchy like:

modules:
  lmod:
    hierarchical_scheme:
      - lapack # Adds lapack on top of MPI/Compiler

with the only limitation that there is no support for packages providing two services (in this case both MPI and LAPACK). This will be solved here.

adamjstewart · 2017-02-19T17:53:30Z

Sounds good to me. I did a quick run-through of all open Issues and PRs and labeled any that I thought were related to modules:
https://github.com/LLNL/spack/labels/modules

citibeth · 2017-02-20T03:57:16Z

I like StringTemplate because it's the only template engine I've seen that does sub-templates gracefully, rather than dumping you on the floor. The core observation is that templating is a good fit for functional languages. I have no idea if StringTemplate would be practial for use in Spack.
Simply including jinja2 in Spack (or whatever template engine you decide to use) is goign to be better than building your own. I agree, we should just include this stuff in Spack for now, and fix the bootstrapping issues two generations later.

@citibeth spack module loads and spack setup commands

I don't have much interest in this. I'm happy with how my hacky "Python templates" are working there. However, if someone else wants to change the way the templating is done, and it doesn't break the current usage, I wouldn't be against it. I suppose it would make things more flexibility; but so far, I haven't come across a need for that flexibility in the spconfig.py file.
I don't expect this PR will affect Package-Level Module Autoloading #3134 particularly much, other than the usual needs to rebase / merbe.

luigi-calori · 2017-02-20T11:57:45Z

First of all thanks a lot for the great work on modules. Using template for module generation was something I had in mind since a while and now here it is ;-)
I' ve had a very quick look at the code, some questions:

Would the user be able to override the predefined templates?
In our current custom modules deployment, some modules require specific code to be added.
would it be possible to define a template that expand a custom field that is declared in some package.py file. Otherwise, the custom code should be specified in an include file defined in module.yaml
On our cluster we have old, pure tcl module setup. In order to ease spack adoption:
Our module system do not have modulecmd, so we can not use modules loading in package.yaml
we addressed install fails looking for modulecmd #2924 by resurrecting and modifying New modulecmd detection algorithm. #2426 . I have an WIP PR for that:
Ok to submit?
We currently use some module grouping ( profile ) to reduce the number of entries presented by
module av . Is there a plan to support module path grouping based on some custom info,
either added in modules.yaml or in individual package.py

citibeth · 2017-02-20T13:24:28Z

Also, we would need to get #2664 merged before this PR messes with spack setup templates.

alalazo · 2017-02-20T13:30:39Z

@citibeth I don't plan to touch commands here.

alalazo · 2017-02-20T14:42:06Z

@luigi-calori

Would the user be able to override the predefined templates?

The mechanism currently implemented permits you to specify a list of template_dirs in the config.yaml file like:

config:
  # Locations where templates should be found
  template_dirs:
    - $spack/templates

To override templates for module files you have then two possibilities.

The first one is at package level, if it makes sense to do so. In this case you need to set e.g. the tcl_template attribute to override the template for tcl.

The second possibility (with higher precedence) is to specify the template in the modules.yaml file under an anonymous spec entry. See the unit tests for more insight on this.

In our current custom modules deployment, some modules require specific code to be added.
would it be possible to define a template that expand a custom field that is declared in some package.py file. Otherwise, the custom code should be specified in an include file defined in module.yaml

You can use all the powers of jinja2 and the mechanisms above to define custom code for your templates.

On our cluster we have old, pure tcl module setup. In order to ease spack adoption:
Our module system do not have modulecmd, so we can not use modules loading in package.yaml
we addressed #2924 by resurrecting and modifying #2426 . I have an WIP PR for that:
Ok to submit?

It's always ok to submit a PR. Maybe you want to ask people in #2924 for an update on the state of things. Do you think that work will be related to this refactoring?

We currently use some module grouping ( profile ) to reduce the number of entries presented by
module av . Is there a plan to support module path grouping based on some custom info,
either added in modules.yaml or in individual package.py

Start using lmod! 😄

alalazo · 2017-02-22T14:53:30Z

@TheQueasle I think you may be interested in this PR as it should extend the functionality introduced in #2902 . Can you tell me if it fits your use case?

TheQueasle · 2017-02-22T15:07:47Z

@alalazo Cursory glance things look good. I'll need to carve out some time to play around with this to be 100% sure.

Honestly though, so long as I can inject arbitrary code in the generated modulefiles it will suit me just fine, and it looks like you covered that usecase already. 😄

alalazo · 2017-02-25T13:16:32Z

@adamjstewart The latest commit should address the change in behavior for spack module refresh when called without specifying the module type. I also changed spack module rm for consistency.

adamjstewart · 2017-02-25T14:40:44Z

lib/spack/spack/cmd/common/arguments.py

    choices=spack.modules.module_types.keys(),
-    default=spack.modules.module_types.keys()[0],
+    action='append',
    help='type of module files [default: %(default)s]')


What's your plan with the default? Also, if you change the default, make sure to change the help message.

Good catch on the help message. I changed the action to append so that one can do:

spack module refresh -m tcl -m lmod ...

The default is the one you proposed (all the module files that are known) except that it's set in the method module.

alalazo · 2017-03-02T15:23:47Z

@tgamblin @adamjstewart @becker33 @scheibelp and anybody interested: I think this PR is ready for a first review round. I'll be waiting for questions / comments.

The correct place to set the mutual references between spec and package objects at the end of concretization. After a call to concretize we should now be ensured that spec is the same object as spec.package.spec. Code in `build_environment.py` that was performing the same operation has been turned into an assertion to be defensive on the new behavior.

spack#3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies.

Added an entry in `config.yaml` (`template_dirs`) to list all the directories where Spack could find templates for `jinja2`. Module file generators have a simple override mechanism to override template selection ('modules.yaml' beats 'package.py' beats 'default').

Common fixtures related to module file generation have been extracted in `conftest.py`. All the mock configurations for module files have been extracted from python code and have been put into their own yaml file. Added a `context_property` decorator for the template engine, to make it easy to define dictionaries out of properties. The default for `verbose` in `modules.yaml` is now False instead of True.

The contexts that are used in conjunction with `jinja2` templates to generate module files can now be extended from package.py and modules.yaml. Module files generators now infer the short description from package.py docstring (and as you may expect it's the first paragraph)

`module refresh` without `--module-type` specified tries to regenerate all known module types. The same holds true for `module rm` Configure options used at build time are extracted and written into the module files where possible.

Added test for exceptional paths of execution when generating Lmod module files. Fixed a few compatibility issues with python3. Fixed a bug in Tcl with naming_scheme and autoload + unit tests

The reference section for module files has been reorganized. The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml Also: - Licenses have been updated to include NOTICE and extended to 2017 - docstrings have been reformatted according to Google style

All the callbacks in `RPackage` and `WafPackage` that are not build phases have been modified not to accept a `spec` and a `prefix` argument. This permits to leverage the common `configure_args` signature to insert by default the configuration arguments into the generated module files. I think it's preferable to handling those packages differently than `AutotoolsPackage`. Besides only one package seems to override one of these methods.

Fixed broken indentation in `spack module refresh` (probably a rebase gone silently wrong?). Filter the writers for blacklisted specs before searching for name clashes. An error with a single writer will not stop regeneration, but instead will print a warning and continue the command.

spack#3183 removed the print_help function from the "modules" module. This adds it back so that if a user invokes 'spack load foo' without having sourced an environment setup script, they will be prompted to do so.

Fixes #6126 #3183 removed the print_help function from the "modules" module. This adds it back so that if a user invokes 'spack load foo' without having sourced an environment setup script, they will be prompted to do so. This also improves the placeholder message to tell the user to invoke 'spack' as shell function rather than as an executable.

alalazo added modules refactoring tests General test capability(ies) WIP labels Feb 19, 2017

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from 58882bc to 1b48cdd Compare February 19, 2017 15:24

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from 7e15211 to 08b9f16 Compare February 20, 2017 14:27

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch 2 times, most recently from 39db11f to a4e5104 Compare February 22, 2017 11:30

alalazo mentioned this pull request Feb 22, 2017

ValueError: Specs passed to a DirectoryLayout must be concrete! #2911

Closed

alalazo mentioned this pull request Feb 22, 2017

Add customization of env modules short description #1114

Closed

paulhopkins mentioned this pull request Feb 23, 2017

Enable module autoload and standard layout by default #3224

Closed

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from d7aa049 to ceaac9c Compare February 23, 2017 19:27

alalazo mentioned this pull request Feb 24, 2017

Module naming scheme OPTIONS and the emacs package #2884

Closed

adamjstewart reviewed Feb 25, 2017

View reviewed changes

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from 3be6ab0 to 89498e4 Compare March 2, 2017 09:14

alalazo added the RFC label Mar 2, 2017

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from 63b0ca3 to 0d113fd Compare March 16, 2017 16:39

alalazo mentioned this pull request Mar 30, 2017

Enhanced Default CMake Package #3626

Closed

alalazo mentioned this pull request Aug 22, 2017

Use compilers for the same Spack #5101

Closed

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch 2 times, most recently from 19fed1f to 07ae702 Compare September 11, 2017 07:20

alalazo mentioned this pull request Sep 12, 2017

Spec.concretize() sets mutual spec-package references #5339

Merged

alalazo added 11 commits September 12, 2017 13:51

Added jinja2 and MarkupSafe to vendored packages.

8157d4d

'module refresh' regenerates all modules by default

7cdd776

`module refresh` without `--module-type` specified tries to regenerate all known module types. The same holds true for `module rm` Configure options used at build time are extracted and written into the module files where possible.

Fixed python3 compatibility, tests for Lmod and Tcl.

1b580de

Added test for exceptional paths of execution when generating Lmod module files. Fixed a few compatibility issues with python3. Fixed a bug in Tcl with naming_scheme and autoload + unit tests

alalazo force-pushed the refactoring/modulefiles_generated_with_a_template_engine branch from 07ae702 to 38bc977 Compare September 12, 2017 15:57

This was referenced Sep 13, 2017

Spack module command uses installed pkg repo #5297

Closed

Some issues with module bootstrapping #5351

Closed

becker33 merged commit b1d129e into spack:develop Sep 19, 2017

alalazo deleted the refactoring/modulefiles_generated_with_a_template_engine branch September 19, 2017 19:36

alalazo mentioned this pull request Sep 20, 2017

modules: Do not force suffixes to be separated by '-' #1295

Closed

pramodskumbhar mentioned this pull request Sep 26, 2017

paraview: openmpi spec not constructed properly during install? #3581

Closed

alalazo mentioned this pull request Sep 30, 2017

Make setup use all environment changes from build_environment #5546

Closed

scheibelp mentioned this pull request Nov 3, 2017

Add back help placeholder for "spack load" #6127

Merged

tgamblin added this to the v0.11.0 milestone Nov 12, 2017

alalazo mentioned this pull request Dec 10, 2019

Recommended way to invoke/use spack within larger "scripts" #5100

Closed

Conversation

alalazo commented Feb 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Motivation

Modifications planned (or already done)

Uh oh!

adamjstewart commented Feb 19, 2017

Uh oh!

alalazo commented Feb 19, 2017

About this PR

About lmod support and issues

Uh oh!

adamjstewart commented Feb 19, 2017

Uh oh!

citibeth commented Feb 20, 2017

Uh oh!

luigi-calori commented Feb 20, 2017

Uh oh!

citibeth commented Feb 20, 2017

Uh oh!

alalazo commented Feb 20, 2017

Uh oh!

alalazo commented Feb 20, 2017

Uh oh!

alalazo commented Feb 22, 2017

Uh oh!

TheQueasle commented Feb 22, 2017

Uh oh!

alalazo commented Feb 25, 2017

Uh oh!

adamjstewart Feb 25, 2017

Choose a reason for hiding this comment

Uh oh!

alalazo Feb 26, 2017

Choose a reason for hiding this comment

Uh oh!

alalazo commented Mar 2, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

alalazo commented Feb 19, 2017 •

edited

Loading

About `lmod` support and issues