I haven't had a chance to test this but it looks great (and pretty straightforward) to me. I could see extending this with a syntax to describe different hierarchical schemes (e.g. not just mpi/compiler/etc.). If you can address some of the comments and push changes, I think this is pretty close to being merged into develop with preliminary support.

I'll probably add something to disable/enable certain types of modules in the config file, as well, since Spack now does three types and I'm not sure everyone wants them all.

One more question: Are there edits that need to be made to share/spack/setup-env.[c]sh? Or are the env vars used by tcl modules and Lmod the same?

I could see extending this with a syntax to describe different hierarchical schemes (e.g. not just mpi/compiler/etc.).

If you have already a clear idea in mind, I am available to extend this implementation.

Are there edits that need to be made to share/spack/setup-env.[c]sh? Or are the env vars used by tcl modules and Lmod the same

To my knowledge Lmod is an implementation of Environment Modules with some unique features, thus the variables that need to be set are the same. The line:

_spack_pathadd MODULEPATH "$_sp_share_dir/modules/$SYS_TYPE"

will set-up a configuration where Lmod is used with a "flat" module directory. To use hierarchical modulefiles the same line should read:

_spack_pathadd MODULEPATH "$_sp_share_dir/lmod/modulefiles/Core"

This brings me a couple question:

• Should I change the name of class LmodModule to something like class LuaHierarchicalModule? (It may be more appropriate since Lmod can also deal with TclModule)
• I miss a distinction based on SYS_TYPE in my directory name for hierarchical modules. Should it be added?

I still need to implement:

@property
def use_name(self):
    # FIXME : this needs to be implemented. It seems to be used only in module.module_find(m_type, spec_array)
    pass

What is this function used for?

I could see extending this with a syntax to describe different hierarchical schemes (e.g. not just mpi/compiler/etc.).

If you have already a clear idea in mind, I am available to extend this implementation.

I have some ideas but not enough to constitute a 'spec'. So let's go with your version for now and I can add some configurability later.

Are there edits that need to be made to share/spack/setup-env.[c]sh? Or are the env vars used by tcl modules and Lmod the same

To my knowledge Lmod is an implementation of Environment Modules with some unique features, thus the variables that need to be set are the same. The line:

_spack_pathadd MODULEPATH "$_sp_share_dir/modules/$SYS_TYPE"

will set-up a configuration where Lmod is used with a "flat" module directory. To use hierarchical modulefiles the same line should read:

_spack_pathadd MODULEPATH "$_sp_share_dir/lmod/modulefiles/Core"

Ok cool -- I think this is fine. I can add some configurable options to select between the TCL module and Lmod.

This brings me a couple question:

Should I change the name of class LmodModule to something like class LuaHierarchicalModule? (It may be more appropriate since Lmod can also deal with TclModule)

I think the first one's fine. Most people will want Lmod to be hierarchical.

I miss a distinction based on SYS_TYPE in my directory name for hierarchical modules. Should it be added?

Yes, please.

I still need to implement:

@property
def use_name(self):
   # FIXME : this needs to be implemented. It seems to be used only in module.module_find(m_type, spec_array)
   pass

What is this function used for?

use_name is basically what's used for the filename of the module. It's also what's printed when you do a spack module find, and what gets loaded when you do spack load. Given that you already overrode file_name, I think the right thing to do is to move the code in file_name to use_name so that use_name prints the name without a suffix and file_name has the suffix... Basically use_name should print whatever you'd pass to module load.

I implemented use_name and added self.spec.architecture to the naming scheme. I am not sure anyhow that there won't be some glitches when trying to load or search hierarchical modules directly from spack.

In fact Lmod provides its own command to search for a package:

$ module spider hdf5

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  hdf5:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        hdf5/1.8.13

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  To find detailed information about hdf5 please enter the full name.
  For example:

     $ module spider hdf5/1.8.13
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

$ module spider hdf5/1.8.13

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  hdf5: hdf5/1.8.13
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    This module can only be loaded through the following modules:

      gcc/5.2.0  mpich/3.0.4
      mpich/3.0.4

    Help:
      HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex da
ta. 

and sometimes you may access to a module in multiple ways (as in the example above, where hdf5/1.8.13 is compiled using either default compiler over mpich or gcc 5.2.0 over mpich)

I took a look at this, there's a lot of different things going on here at the same time:

• support for using Lmod as a modules tool
• support for using a hierarchical module naming scheme
• support for using module files in Lua syntax

I'm not sure whether you want to be doing everything at the same time (although the combination sure makes sense).

As @tgamblin already mentioned, this also hard-codes to the traditional Core/Compiler/MPI hierarchy, while having something more flexible would be nice.
In EasyBuild, we implement support the hierarchical modules via a custom module naming scheme, for example: https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/module_naming_scheme/hierarchical_mns.py .
But, you can easily tweake that slightly to your needs, for example by adding an extra hierarchy level for BLAS or whatever, by copying and modifying that module slightly, and then configuring EB to pick it up and use it.

The support for using Lmod as a modules tool (i.e., to use lmod python <command> rather than modulecmd python <command>), and to generate module files Lua syntax, is handled entirely separately from each other in EasyBuild.
Although you (kind of) need to use Lmod when doing a hierarchy, you don't need to use module files in Lua syntax; you can do it in Tcl syntax just fine if you want to (for example, but avoid 'vendor locking' to using Lmod).
Vice versa, you can also use Lmod without using a module hierarchy.

Since I was involved with much of the support for using Lmod, generating module files in Lua syntax and supporting hierarchical modules in EasyBuild, I'm aware of the hoops you need to make Spack jump through. I'm happy to answer any questions. The paper on using Lmod and a module hierarchy I'm a co-author of is a good read to get a good view on this, see http://hpcugent.github.io/easybuild/files/hust14_paper.pdf .

@boegel : many thanks for taking the time to review and comment on this PR.

I took a look at this, there's a lot of different things going on here at the same time:

• support for using Lmod as a modules tool
• support for using a hierarchical module naming scheme
• support for using module files in Lua syntax

I'm not sure whether you want to be doing everything at the same time (although the combination sure makes sense).

This changeset addresses only two of the three aspects you cited, that is the support for using a hierarchical module naming scheme and for writing module files in Lua syntax.

I merged together those two aspects mainly because:

1. Lua seems to be the native way to write module files in Lmod
2. to make full use of the features provided by Lmod you need to extend anyhow the TCL modulefiles syntax with the family function (or at least this is how I got it from Lmod documentation)

Do you see relevant use cases where it would it make sense to use Lmod with "traditional" TCL modulefiles, or where one would use modulecmd with a hierarchical layout?

As @tgamblin already mentioned, this also hard-codes to the traditional Core/Compiler/MPI hierarchy, while having something more flexible would be nice.
In EasyBuild, we implement support the hierarchical modules via a custom module naming scheme, for example: https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/module_naming_scheme/hierarchical_mns.py .
But, you can easily tweake that slightly to your needs, for example by adding an extra hierarchy level for BLAS or whatever, by copying and modifying that module slightly, and then configuring EB to pick it up and use it.

I agree that having more flexibility would be nice, but I don't see an easy way at the moment to go far ahead of what is in this PR. As I see it 'levels' in a hierarchical naming scheme maps directly to virtual dependencies (you have 'compiler', 'mpi', 'blas', etc.). Having the traditional Core/Compiler/MPI hierarchy covers a good part of our use cases, and is pretty straightforward. Adding other things to the mix (like 'blas', 'lapack', 'cuda', etc.) naturally calls for a better support of a dependency matrix, where things should be visible only if providers of all their virtual dependencies are loaded. As far as I understood this is still an open area of research for Lmod. How do you deal in EasyBuild with this case?

The support for using Lmod as a modules tool (i.e., to use lmod python rather than modulecmd python ), and to generate module files Lua syntax, is handled entirely separately from each other in EasyBuild.

I guess module files generation and support for the delegation to a module tool (e.g. spack load and similar commands) are separated by design also in spack, but @tgamblin may obviously say more about that. I don't provide support for Lmod as a tool in this PR, but I am confident it is something that will be addressed in the near future.

@tgamblin Do you think this PR misses anything to be merged?

@tgamblin Odd : this old PR seems not to be able to trigger Travis anymore...

Even when you rebase it?

@adamjstewart Yes

@tgamblin Any chance this could be reviewed manually ?

Huh, Travis works again for you too eh?

Not really... it seems it's pulling a travis.yaml file from a while ago...