Bootstrap environment-modules by krafczyk · Pull Request #3057 · spack/spack

krafczyk · 2017-02-07T20:29:15Z

This branch adds the ability to automatically build and enable environment-modules as part of spack-env.sh. The added code detects whether either dotkit or modules is available, and if neither is available, environment-modules is built and module function is introduced.

This change makes spack more portable as module support is automatically setup.

davydden

I am not the best guy to review this, but I like the feature! On my mac I always install environment-modules myself anyway. Same with ubuntu.

citibeth · 2017-02-07T20:41:58Z

Nice idea, but I think this is the wrong place to do it. We've tossed around the idea of Spack bootstrapping before. If we start down that path, I expect the bootstrapping procedure will grow to be more sophisticated (and slower) over time. See #2020. I don't think it should be run, in general, in the user's .bashrc.

I would be amenable to a new spack bootstrap command that, when run, installs all sorts of things it thinks you need but don't have. I would like to see the bootstrapping logic centralized, so we can manage it (rather than being scattered across a dozen places in Spack).

Would you be interested in adding a spack bootstrap command whose first function is to look for environment-modules and install if it's not on your system?

krafczyk · 2017-02-07T21:03:48Z

Part of the problem I see with the code I introduced being moved to a spack bootstrap command is that the module needs to be made available as part of the spack-env.sh script for greatest ease of use.

However, that said, a spack bootstrap kind of command would be good to automatically build all necessary libraries as described in #2020 which may grow to include more than just environment-modules. environment-modules is the only package I know about which spack needs to run effectively which isn't on most computers.

How about this:

I can add a basic spack bootstrap which handles installing environment-modules and others as necessary
I cut out the stuff which installs environment-modules from spack-env.sh, but keep the stuff that detects whether module is missing and attempts to use one inside spack, then produces a warning message if module wasn't available and wasn't installed along with the suggestion that they run spack bootstrap to make it available and run spack-env.sh again to get module.

This effectively separates the process of setting up any basic stuff that spack needs along with giving spack-env.sh the ability to make the module command available if it isn't already.

citibeth · 2017-02-07T21:08:43Z

How about this: 1. I can add a basic spack bootstrap which handles installing environment-modules and others as necessary 2. I cut out the stuff which installs environment-modules from spack-env.sh, but keep the stuff that detects whether module is missing and attempts to use one inside spack, then produces a warning message if module wasn't available and wasn't installed along with the suggestion that they run spack bootstrap to make it available and run spack-env.sh again to get module. I think this means that `spack-env.sh` would detect if

`environment-modules` is missing, but not try to build it. Sounds good to me. Remember also that not everyone uses `environment-modules`. Some use `lmod`. Any changes to `spack-env.sh` would have to be compatible with both.

alalazo · 2017-02-07T21:09:16Z

@krafczyk Did you consider building lmod instead of environment-modules? It's completely backward compatible with TCL module files and is currently maintained.

citibeth · 2017-02-07T21:22:28Z

...and requires the installation of Lua, for which we don't really have proper Spack support.

…

On Tue, Feb 7, 2017 at 4:09 PM, Massimiliano Culpo ***@***.*** > wrote: @krafczyk <https://github.com/krafczyk> Did you consider building lmod instead of environment-modules? It's completely backward compatible with TCL module files and is currently maintained. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#3057 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd2ypKj1Ex8fvJ2sG27ynHJUH0txVks5raN19gaJpZM4L5_l5> .

krafczyk · 2017-02-07T21:31:40Z

@alalazo, I didn't, but simply because I'm more familiar with environment-modules. I suppose we could install that too.

luigi-calori · 2017-02-08T10:58:45Z

Thank you a lot: I needed exactly that for bootstrap on my laptop Mint, tested, it seems to work.
One issue about current implementation is that it installs env modules at first
source share/spack/setup-env.sh that, in my workflow, comes before configuration setup.
Think that your improvement following from @citibeth comment would help also my workflow:

my deployment script could then be:
source share/spack/setup-env.sh

source share/spack/setup-env.sh; spack bootstrap

krafczyk · 2017-02-08T17:05:25Z

@luigi-calori What I was considering was more like this:

source share/spack/setup-env.sh; spack bootstrap; source share/spack/setup-env.sh;

Would that be reasonable to you?

luigi-calori · 2017-02-08T17:38:18Z

@krafczyk Sure,
that was what I was had intended.
to summarize, starting from a clean system:

clone spack
source share/spack/setup-env.sh
edit *.yaml add compilers, add mirror
spack bootstrap
source share/spack/setup-env.sh

At this point, module is available, and it will be for all the following session where
source share/spack/setup-env.sh is executed.
This could also be added to .bashrc

krafczyk · 2017-02-08T17:53:10Z

As it turns out, spack bootstrap is already a command which clones spack into a new prefix. Any suggestions for a new command name? I can either rename the current spack bootstrap to something else and use boostrap for this purpose, or I can create a new command name for this purpose. Ideas?

adamjstewart · 2017-02-08T17:57:47Z

@tgamblin Does anyone even use spack bootstrap?

citibeth · 2017-02-08T18:17:32Z

I second @adamjstewart here. Let's rename the old `spack bootstrap` and reuse the name.

krafczyk · 2017-02-08T18:36:39Z

Alright, how about spack clone for a new name for the old spack bootstrap?

citibeth · 2017-02-08T19:03:22Z

Sounds fine with me. I doubt the old command is used much, so it doesn't really matter what it's called.

…

On Wed, Feb 8, 2017 at 1:36 PM, Matthew Scott Krafczyk < ***@***.***> wrote: Alright, how about spack clone for a new name for the old spack bootstrap? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3057 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd4ZZ9lrokGny9OjA8g55_ZzJzwpAks5rags4gaJpZM4L5_l5> .

krafczyk · 2017-02-08T21:49:52Z

Alright folks, I've moved bootstrap functionality to clone, and I've turned bootstrap into a command to install necessary packages. I've also changed the logic of spack-env.sh so that it doesn't automatically install the missing packages and only activates module if you don't already have it available and you have the right package already installed.

As far as spack bootstrap goes, I've decided to go the simple route for now and just install a list of specs which we deem all spack repos ought to have. I tried to figure out how to determine if module was defined in the containing shell environment but it seems like there's no way to do that. If someone can come up with a better way in the future that'd be great. For now I think this is all the functionality I really want so I'd love to get this merged.

krafczyk · 2017-02-08T21:52:38Z

In addition, I duplicated a few arguments and functionality from install. I'd like to get comments on how I've implemented it and whether there might be a better way.

adamjstewart · 2017-02-08T22:18:45Z

lib/spack/spack/cmd/bootstrap.py

-description = "create a new installation of spack in another prefix"
+from spack.util.executable import which

+description = "bootstrap packages needed for spack to run smoothly."


Descriptions should not end in a period to match what argparse prints by default.

Fixed below.

citibeth

One big question: Once environment-modules has been installed, how will Spack change its configuration to use it?

citibeth · 2017-02-08T22:24:02Z

lib/spack/spack/cmd/bootstrap.py

+        help="do not check packages against checksum")
+    subparser.add_argument(
+        '-v', '--verbose', action='store_true', dest='verbose',
+        help="display verbose build output while installing")


Should these be shared with spack install?

We need to move them into common_arguments since they're used in diy as well. I'd like to do that in a separate PR though to keep complexity down.

Once the user has executed spack bootstrap They need to run spack-env.sh again in order to allow that script to define the module function. I've found that what I have in spack-env.sh seems to work perfectly.

We need to move them into common_arguments since they're used in diy as well.

OK, let's just leave it as-is. Common arguments should probably go with #2664.

krafczyk · 2017-02-09T17:41:15Z

CI tests passed, but one coverage test failed. Can someone tell me what the problem is?

If nobody has any objections, I'd like to get this merged.

bd4 · 2017-02-09T22:18:43Z

I'm an experienced Linux user (since 1998) and I was pretty confused initially about how to make use of the software I had installed via spack. I like the idea of having a module solution automatically bootstrapped and activated along with shell support, but another option would be to update the installation doc to recommend installation of either lmod or environment-modules (currently it's buried fairly deep and it was not easy for me to find). A doc update is worth considering in either case I think.

I've included suggestions from scheibelp.

Make bootstrap command accept a list of packages which may satisfy requirements.

- Make comments more clear - Add user suggestions about modules - Make searching for required packages about modules smarter.

krafczyk · 2017-09-05T20:05:02Z

@scheibelp I've tried to incorporate your changes with some new commits. Please have a look.

scheibelp

Thanks a lot for the updates. I have a few additional requests:

I have a suggestion for simplifying the check for environment-modules in bootstrap (I apologize for not concretely explaining my original request - the comment includes a concrete example)
There are 2 documentation update requests

scheibelp · 2017-09-05T20:39:34Z

lib/spack/docs/getting_started.rst

+     try the following procedure:

-#. Install with:
+      * Consider using system tcl (as long as your system has Tcl version 8.0 or later):


It has occurred to me that this consideration would apply whether you install environment-modules with spack install or spack bootstrap. IMO it makes sense to reorganize like:

Install environment-modules (with spack install or spack bootstrap). You may need to use system tcl...

Update path to use spack-installed environment-modules (manually or with setup-env.sh)

So, my comment about tcl is simply because tcl takes the longest to build and install in environment-module's dependency stack. So since it is a common package to have already installed, I mention that you can get spack to use it instead of rebuilding it.

However.. Maybe this isn't a good idea to suggest because spack currently does not have a good method of interacting with native installed packages. Maybe I should just strike that bit about tcl entirely, thoughts?

It looks like the comment about using system tcl was originally added by @citibeth. IMO it is worth keeping, but I think it should be added at the end of the section (especially if the only purpose is to speed up the installation):

You can speed up installation of environment-modules by referring Spack to use a system-installed tcl if it is available...

I'm not sure what you mean by:

Maybe this isn't a good idea to suggest because spack currently does not have a good method of interacting with native installed packages.

Are you saying in your experience that specifying externals in packages.yaml has caused problems for you?

No, the mechanics of adding externals seems to be okay for now.

What I worry about is a system /usr/lib contaminating the link line somewhere and linking against system packages on accident where spack packages should be used. I don't yet have a specific example when this has caused problems yet, but I feel that perhaps for stuff we're claiming will work well for all spack users we shouldn't loudly proclaim this functionality when it may cause some problems in some cases.

There has been work to prevent system paths like /usr/lib from making their way into link paths. External packages are already linked in the guide on getting started. So I think it's worth keeping this. I do think it should be put at the end of the section though (not as part of the 3 steps), since it is easier (if longer) not to point Spack to the system tcl.

scheibelp · 2017-09-05T20:41:32Z

lib/spack/docs/module_file_support.rst

   $ source $SPACK_ROOT/share/spack/setup-env.csh

+Executing ``spack bootstrap`` will install the ``environment-modules`` package
+and ``bash`` and ``ksh`` users sourcing the appropriate setup file will have


I'd prefer:

When bash and ksh users update their environment with setup-env.sh, it will check for spack-installed environment modules and add the module command to their environment; this only occurs if the module command is not already available. You can install environment-modules with spack-bootstrap as described in (link to section)

This deals with a number of minor issues. For example it doesn't draw the user's attention to spack bootstrap unless they are using bash or ksh, which I think is worthwhile, since csh users don't benefit from spack bootstrap.

Also, from what I can tell everything in the following section "spack load / unload" applies just the same when setup-env.sh is invoked after spack bootstrap, but I wanted to confirm that with you.

Yes that is correct.

I like what you wrote better, I'm changing it.

scheibelp · 2017-09-05T21:13:48Z

lib/spack/spack/cmd/bootstrap.py

+        for req in requirement_list:
+            spec_req = spack.Spec(req)
+            spec_req.concretize()
+            installed_specs = spack.store.db.query(spec_req)


I apologize but I did not thoroughly explain myself. This achieves what I was looking for but I think it can be done simpler: you should be able to db.query(requirement). For example you can db.query('environment-modules') and it will match any installation of environment modules regardless of variant settings; you don't have to query for both possible settings of the X variant. So to make it concrete the dict would look like:

requirement_dict = {'environment-modules': 'environment-modules~X'}

and the idea would be that you query for environment-modules but if no such installed package is available, you install environment-modules~X.

Let me know if that is unclear.

Okay, that makes more sense. I'll update the code to take this into account.

I assume that my use of spack.Spec is correct?

krafczyk · 2017-09-06T18:22:41Z

@scheibelp I've committed some changes taking your review into account. Have a look.

scheibelp · 2017-09-06T21:24:09Z

lib/spack/docs/getting_started.rst

+build it for you!

-#. Consider using system tcl (as long as your system has Tcl version 8.0 or later):
+What follows are three steps describing how to install and use environment-modules with spack.


This is two steps now isn't it (with your most recent changes)?

The third step is to test that the module command works properly.
Do you think I should do away with that step?

Sorry I missed that - nope I see 3 and I like the 3rd step for verification.

krafczyk · 2017-09-07T15:42:38Z

@scheibelp I've restored the tcl discussion to the docs.

scheibelp · 2017-09-07T21:25:33Z

lib/spack/docs/getting_started.rst

+#. Install ``environment-modules``.

-      .. code-block:: yaml
+   * Spack can build and install ``environment-modules`` for you.


IMO this wording is slightly awkward since spack is building environment-modules for you in both cases. Perhaps:

spack bootstrap will build environment-modules for you (and may build other packages that are useful to the operation of Spack).

Other than that LGTM - make that change and I'll merge this.

krafczyk · 2017-09-07T21:34:11Z

@scheibelp I've gone and made that change

scheibelp · 2017-09-08T02:58:21Z

Thanks! Sorry but it appears there was a spurious test failure - could you close and reopen this?

scheibelp · 2017-09-08T03:25:29Z

Thanks! Sorry but it appears there was a spurious test failure - could you close and reopen this?

NM I can do it myself: I'll merge it in the morning.

scheibelp · 2017-09-08T17:15:30Z

Thanks! I appreciate all the edits

krafczyk · 2017-09-08T17:59:00Z

@scheibelp Thanks a lot for your help here! I'm so glad this is finally merged.

krafczyk requested review from adamjstewart, citibeth, davydden and tgamblin February 7, 2017 20:29

davydden reviewed Feb 7, 2017

View reviewed changes

krafczyk force-pushed the features/standalone-spack branch from 0343388 to 3a632b1 Compare February 8, 2017 21:46

krafczyk added WIP feature A feature is missing in Spack labels Feb 8, 2017

adamjstewart reviewed Feb 8, 2017

View reviewed changes

citibeth changed the title ~~Features/standalone spack~~ Bootstrap environment-modules Feb 8, 2017

citibeth reviewed Feb 8, 2017

View reviewed changes

krafczyk force-pushed the features/standalone-spack branch from 4d1dbd2 to 570b215 Compare February 9, 2017 16:06

krafczyk added 3 commits September 5, 2017 15:00

Documentation Update

42d5ad2

I've included suggestions from scheibelp.

Update bootstrap command

9688938

Make bootstrap command accept a list of packages which may satisfy requirements.

Update setup-env.sh

7c2a9f0

- Make comments more clear - Add user suggestions about modules - Make searching for required packages about modules smarter.

scheibelp requested changes Sep 6, 2017

View reviewed changes

krafczyk added 2 commits September 6, 2017 11:17

Simplify bootstrap

6a055ac

Update documentation again

0d2b396

scheibelp reviewed Sep 6, 2017

View reviewed changes

Add tcl statement to docs

d257452

scheibelp reviewed Sep 7, 2017

View reviewed changes

Update docs with more clear bootstrap explanation

dcd986b

scheibelp closed this Sep 8, 2017

scheibelp reopened this Sep 8, 2017

scheibelp approved these changes Sep 8, 2017

View reviewed changes

scheibelp merged commit 51828dd into spack:develop Sep 8, 2017

krafczyk deleted the features/standalone-spack branch September 8, 2017 18:04

scheibelp mentioned this pull request Sep 9, 2017

Revert "Bootstrap environment-modules" #5316

Closed

tgamblin mentioned this pull request Sep 12, 2017

Some issues with module bootstrapping #5351

Closed

scheibelp mentioned this pull request Sep 13, 2017

Remove automatic definition of 'module' command in setup-env.sh #5352

Closed

alalazo mentioned this pull request Sep 16, 2017

setup-env.sh not adding lmod files to MODULEPATH #5377

Closed

krafczyk mentioned this pull request Oct 25, 2017

Auto-Detect Module Systems #2981

Closed

alalazo mentioned this pull request Nov 22, 2017

Auto-Bootstrap Procedure #2020

Closed

adamjstewart mentioned this pull request Mar 25, 2018

Making the SPACK_SHELL variable more robust #7475

Closed

Conversation

krafczyk commented Feb 7, 2017

Uh oh!

davydden left a comment

Choose a reason for hiding this comment

Uh oh!

citibeth commented Feb 7, 2017

Uh oh!

krafczyk commented Feb 7, 2017

Uh oh!

citibeth commented Feb 7, 2017 via email

Uh oh!

alalazo commented Feb 7, 2017

Uh oh!

citibeth commented Feb 7, 2017 via email

Uh oh!

krafczyk commented Feb 7, 2017

Uh oh!

luigi-calori commented Feb 8, 2017

Uh oh!

krafczyk commented Feb 8, 2017

Uh oh!

luigi-calori commented Feb 8, 2017

Uh oh!

krafczyk commented Feb 8, 2017

Uh oh!

adamjstewart commented Feb 8, 2017

Uh oh!

citibeth commented Feb 8, 2017 via email

Uh oh!

krafczyk commented Feb 8, 2017

Uh oh!

citibeth commented Feb 8, 2017 via email

Uh oh!

krafczyk commented Feb 8, 2017

Uh oh!

krafczyk commented Feb 8, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

citibeth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krafczyk commented Feb 9, 2017

Uh oh!

bd4 commented Feb 9, 2017

Uh oh!

krafczyk commented Sep 5, 2017

Uh oh!

scheibelp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!