modules.py: adding 'include' logic to TCL module generation. by TheQueasle · Pull Request #2902 · spack/spack

TheQueasle · 2017-01-22T23:18:07Z

Enable arbitrary TCL code to be included in the module files via the modules.yaml config file.

For example, I want to include /shared/modulefiles/include/logging.tcl on every TCL modulefile generated via spack.

modules.yaml:

modules:
  tcl:
    all:
      include:
        - '/shared/modulefiles/include/logging.tcl'

Module file generated after fake installing htop:

[vagrant@localhost spack]$ spack install --fake htop
==> Installing htop
==> Building htop [Package]
==> Successfully installed htop
  Fetch: .  Build: 0.04s.  Total: 0.04s.
[+] /home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd
[vagrant@localhost spack]$ cat /home/vagrant/spack/share/spack/modules/linux-centos7-x86_64/htop-2.0.0-gcc-4.8.5-7jzqvbb
#%Module1.0
## Module file created by spack (https://github.com/LLNL/spack) on 2017-01-22 22:46:47.997359
##
## [email protected]%[email protected] arch=linux-centos7-x86_64 -7jzqvbb
##
module-whatis "htop @2.0.0"

proc ModulesHelp { } {
puts stderr "This is htop, an interactive process viewer for Unix systems. It is a"
puts stderr "text-mode application (for console or X terminals) and requires ncurses."
}

prepend-path PATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/bin"
prepend-path CMAKE_PREFIX_PATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/"
prepend-path LIBRARY_PATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/lib"
prepend-path LD_LIBRARY_PATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/lib"
prepend-path CPATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/include"
prepend-path MANPATH "/home/vagrant/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/htop-2.0.0-7jzqvbbs5m3t6u5m2sovnbxkqz7iargd/man"
source /shared/modulefiles/include/logging.tcl

Currently, the logic only checks if the file to include exists, and fails if it does not.

[vagrant@localhost spack]$ sudo rm /shared/modulefiles/include/logging.tcl
[vagrant@localhost spack]$ spack module refresh --module-type tcl --delete-tree -y
==> Regenerating tcl module files
==> Error: Include file must exist!** You may want to check your `modules.yaml` configuration file **

Module generation aborted.

I chose the word include for use in modules.yaml because that is what I thought the TCL command was called when I started this feature...but after testing I discovered it is source. I'm not sure if I like include or source better for the config yaml...

I brought up this feature request during SC16 at one of the Spack talks (which were all awesome btw) and no one seemed too opposed to it. So I finally decided to sit down and attempt to add the feature.

I'm pretty new to Python development, so I'm sorry for any glaring issues you see with this PR 😑

…arbitrary TCL code to be included in the module files

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

* modulefiles: are now generated using a template engine refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies * template dirs in config.yaml + override mechanism for pkgs Modifications: - added the entry 'template_dirs' to config.yaml - added an override mechanism for templates ('modules.yaml' beats 'package.py' beats 'default') - added unit tests * modules: improved test layout and readability Modifications: - factored common fixtures in conftest.py - extracted configurations from tests and put them into yaml files - fixed 'verbose' keyword in modules.yaml and corresponding tests * simplified template engine file layout Modifications: - added a 'context_property' decorator - simplified file layout for tengine (python module instead of package) - added tests for tengine.Context * contexts can be extended from packages and modules.yaml * short description it's the first paragraph of docstrings * module refresh: changed behavior when no module type is specified Modifications: - spack module refresh without `--module-type` specified tries to regenerate all known module types - spack module rm without `--module-type` specified tries to delete all known module types - added tests for the new behavior * module file: added configure options to module files if applicable * lmod: added tests on exceptional paths of execution * python3: fixed issues with python2 vs python3 compatibility * jinja2: added external package * tcl: fixed bugs with naming_scheme and autoload. Added regression tests. * markupsafe: added external package * module tutorial: updated docs, fixed a few bugs caught in the meanwhile. * reference docs for modules: reorganized section The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml

refers spack#2902 refers spack#3173 Modifications in detail: - hooked `jinja2` into spack (but still need to fix the facade) - refactored `modules.py` into the package `spack/modules` - refactored `test/modules.py` accordingly to the item above - added templates for the three kind of module files we generate - extended lmod to multi-providers and deeper hierarchies

spack#3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies.

* Module files now are generated using a template engine refers #2902 #3173 jinja2 has been hooked into Spack. The python module `modules.py` has been splitted into several modules under the python package `spack/modules`. Unit tests stressing module file generation have been refactored accordingly. The module file generator for Lmod has been extended to multi-providers and deeper hierarchies. * Improved the support for templates in module files. Added an entry in `config.yaml` (`template_dirs`) to list all the directories where Spack could find templates for `jinja2`. Module file generators have a simple override mechanism to override template selection ('modules.yaml' beats 'package.py' beats 'default'). * Added jinja2 and MarkupSafe to vendored packages. * Spec.concretize() sets mutual spec-package references The correct place to set the mutual references between spec and package objects at the end of concretization. After a call to concretize we should now be ensured that spec is the same object as spec.package.spec. Code in `build_environment.py` that was performing the same operation has been turned into an assertion to be defensive on the new behavior. * Improved code and data layout for modules and related tests. Common fixtures related to module file generation have been extracted in `conftest.py`. All the mock configurations for module files have been extracted from python code and have been put into their own yaml file. Added a `context_property` decorator for the template engine, to make it easy to define dictionaries out of properties. The default for `verbose` in `modules.yaml` is now False instead of True. * Extendable module file contexts + short description from docstring The contexts that are used in conjunction with `jinja2` templates to generate module files can now be extended from package.py and modules.yaml. Module files generators now infer the short description from package.py docstring (and as you may expect it's the first paragraph) * 'module refresh' regenerates all modules by default `module refresh` without `--module-type` specified tries to regenerate all known module types. The same holds true for `module rm` Configure options used at build time are extracted and written into the module files where possible. * Fixed python3 compatibility, tests for Lmod and Tcl. Added test for exceptional paths of execution when generating Lmod module files. Fixed a few compatibility issues with python3. Fixed a bug in Tcl with naming_scheme and autoload + unit tests * Updated module file tutorial docs. Fixed a few typos in docstrings. The reference section for module files has been reorganized. The idea is to have only three topics at the highest level: - shell support + spack load/unload use/unuse - module file generation (a.k.a. APIs + modules.yaml) - module file maintenance (spack module refresh/rm) Module file generation will cover the entries in modules.yaml Also: - Licenses have been updated to include NOTICE and extended to 2017 - docstrings have been reformatted according to Google style * Removed redundant arguments to RPackage and WafPackage. All the callbacks in `RPackage` and `WafPackage` that are not build phases have been modified not to accept a `spec` and a `prefix` argument. This permits to leverage the common `configure_args` signature to insert by default the configuration arguments into the generated module files. I think it's preferable to handling those packages differently than `AutotoolsPackage`. Besides only one package seems to override one of these methods. * Fixed broken indentation + improved resiliency of refresh Fixed broken indentation in `spack module refresh` (probably a rebase gone silently wrong?). Filter the writers for blacklisted specs before searching for name clashes. An error with a single writer will not stop regeneration, but instead will print a warning and continue the command.

alalazo · 2017-09-20T07:04:20Z

I'll close this one as now you can extend module files with custom templates. Right now, you can find brief examples in the unit tests, e.g. here. In the incoming days I'll start writing more docs for that and other parts of Spack.

If you have questions or remarks feel free to ping me.

modules.py: adding 'include' logic to TCL module generation. Enables …

c7f68ca

…arbitrary TCL code to be included in the module files

davydden added the modules label Jan 26, 2017

adamjstewart requested a review from alalazo February 18, 2017 21:39

alalazo mentioned this pull request Feb 19, 2017

Modulefiles generated with a template engine #3183

Merged

16 tasks

alalazo closed this Sep 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modules.py: adding 'include' logic to TCL module generation.#2902

modules.py: adding 'include' logic to TCL module generation.#2902
TheQueasle wants to merge 1 commit intospack:developfrom
TheQueasle:features/tcl_include_files

TheQueasle commented Jan 22, 2017 •

edited

Loading

Uh oh!

alalazo commented Sep 20, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

TheQueasle commented Jan 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalazo commented Sep 20, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TheQueasle commented Jan 22, 2017 •

edited

Loading