Set default module type based on modules.yaml by adamjstewart · Pull Request #3173 · spack/spack

adamjstewart · 2017-02-17T19:02:17Z

Closes #2910.

Previously, spack module would default to tcl. If you added a modules.yaml like:

modules:
  enable::
    - lmod

spack module refresh would crash. Now, spack module defaults to whatever is in your modules.yaml.

If you have multiple modules enabled, it uses the first. This defaults to tcl on my computer, but since it's a dict, it might be sorted differently for other users. Perhaps we should set the default modules.yaml to enable just tcl instead of tcl + dotkit?

Personally, I would like spack module to default to all enabled modules, not just the first one. But I'm not sure if that makes sense for all of the spack module subcommands. Thoughts? We could allow multiple module types for these commands.

alalazo · 2017-02-17T19:21:59Z

@adamjstewart What I was thinking for this issue is to rename enable to registered_hook (or similar). Then the command:

$ spack module ...

should permit you to choose among all available module types (even the one that are not registered at spack install time). Does it make sense?

adamjstewart · 2017-02-17T19:29:48Z

Idk about renaming, but I do agree that spack module refresh --module-type lmod should work even if lmod isn't in your "enabled" modules.

citibeth · 2017-02-17T19:42:42Z

I believe that the Spack modules UI will be changed; and that what needs to be changed will become clearer with Spack Environments.

adamjstewart · 2017-02-17T20:13:49Z

In the meantime, this change is pretty simple and allows me to run spack module refresh without specifying a --module-type.

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

spack#3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies.

* Module files now are generated using a template engine refers #2902 #3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies. * Improved the support for templates in module files. Added an entry in `config.yaml` (`template_dirs`) to list all the directories where Spack could find templates for `jinja2`. Module file generators have a simple override mechanism to override template selection ('modules.yaml' beats 'package.py' beats 'default'). * Added jinja2 and MarkupSafe to vendored packages. * Spec.concretize() sets mutual spec-package references The correct place to set the mutual references between spec and package objects at the end of concretization. After a call to concretize we should now be ensured that spec is the same object as spec.package.spec. Code in `build_environment.py` that was performing the same operation has been turned into an assertion to be defensive on the new behavior. * Improved code and data layout for modules and related tests. Common fixtures related to module file generation have been extracted in `conftest.py`. All the mock configurations for module files have been extracted from python code and have been put into their own yaml file. Added a `context_property` decorator for the template engine, to make it easy to define dictionaries out of properties. The default for `verbose` in `modules.yaml` is now False instead of True. * Extendable module file contexts + short description from docstring The contexts that are used in conjunction with `jinja2` templates to generate module files can now be extended from package.py and modules.yaml. Module files generators now infer the short description from package.py docstring (and as you may expect it's the first paragraph) * 'module refresh' regenerates all modules by default `module refresh` without `--module-type` specified tries to regenerate all known module types. The same holds true for `module rm` Configure options used at build time are extracted and written into the module files where possible. * Fixed python3 compatibility, tests for Lmod and Tcl. Added test for exceptional paths of execution when generating Lmod module files. Fixed a few compatibility issues with python3. Fixed a bug in Tcl with naming_scheme and autoload + unit tests * Updated module file tutorial docs. Fixed a few typos in docstrings. The reference section for module files has been reorganized. The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml Also: - Licenses have been updated to include NOTICE and extended to 2017 - docstrings have been reformatted according to Google style * Removed redundant arguments to RPackage and WafPackage. All the callbacks in `RPackage` and `WafPackage` that are not build phases have been modified not to accept a `spec` and a `prefix` argument. This permits to leverage the common `configure_args` signature to insert by default the configuration arguments into the generated module files. I think it's preferable to handling those packages differently than `AutotoolsPackage`. Besides only one package seems to override one of these methods. * Fixed broken indentation + improved resiliency of refresh Fixed broken indentation in `spack module refresh` (probably a rebase gone silently wrong?). Filter the writers for blacklisted specs before searching for name clashes. An error with a single writer will not stop regeneration, but instead will print a warning and continue the command.

Set default module type based on modules.yaml

8f2db51

adamjstewart added commands modules labels Feb 17, 2017

tgamblin approved these changes Feb 17, 2017

View reviewed changes

tgamblin merged commit 524303d into spack:develop Feb 17, 2017

alalazo mentioned this pull request Feb 19, 2017

Modulefiles generated with a template engine #3183

Merged

16 tasks

adamjstewart deleted the features/default-module-type branch March 24, 2017 22:10

diaena pushed a commit to diaena/spack that referenced this pull request May 26, 2017

Set default module type based on modules.yaml (spack#3173)

5c44e7e

xavierandrade pushed a commit to xavierandrade/spack that referenced this pull request Jun 16, 2017

Set default module type based on modules.yaml (spack#3173)

f02e7e7

amklinv pushed a commit that referenced this pull request Jul 17, 2017

Set default module type based on modules.yaml (#3173)

722d0d7

healther pushed a commit to electronicvisions/spack that referenced this pull request Jul 26, 2017

Set default module type based on modules.yaml (spack#3173)

039fb4c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set default module type based on modules.yaml#3173

Set default module type based on modules.yaml#3173
tgamblin merged 1 commit intospack:developfrom
adamjstewart:features/default-module-type

adamjstewart commented Feb 17, 2017

Uh oh!

alalazo commented Feb 17, 2017

Uh oh!

adamjstewart commented Feb 17, 2017

Uh oh!

citibeth commented Feb 17, 2017

Uh oh!

adamjstewart commented Feb 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

adamjstewart commented Feb 17, 2017

Uh oh!

alalazo commented Feb 17, 2017

Uh oh!

adamjstewart commented Feb 17, 2017

Uh oh!

citibeth commented Feb 17, 2017

Uh oh!

adamjstewart commented Feb 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants