Comments based on the Docs in the main PR

Environments are created, by default, in $spack/var/spack/environments, and you refer to them by name. You can also create environments external to Spack in directories:

spack env create -d directory-environment
spack env activate -d ./directory-environment   # -d isn't necessary unless ambiguous

If you do all your work in Spack Environments, then it is now possible to garbage collect unnecessary packages. This is important on shared supercomputers with inode limits. BUT... Spack has to know about all the environments it is supposed to keep around. Creating environments outside of Spack will break garbage collection (as would using Spack without environments). The idea of creating an environment outside of Spack is an obvious feature; but I would want to have a clear idea of why we need it???

In any case, this downside of this feature should be documented.

Also, you don't have to activate them to use them. Spack has a new
-e / --env option you can use to execute any Spack command in a
specific environment:

This should have been said right at the top with spack activate. Also, don't bother saying the feature is "new;" all of Spack Environements are new.

spack.yaml describes the specs you want in your environment. It's created when you do spack env create <name>, and it describes the specs you've added so far:

I like that the former contents of the config/ sub-directory have been merged into the spack.yaml file. That's a better design. Now... is it possible to use a single YAML file for a config, rather than separate packages.yaml, etc? That would be nice.

This is what other project dependency managers call a manifest file --
a list of the things you want to install with a project.

It might be better to just describe Spack Environments; and then in a later place, relate Spack Environments to how other package managers do environments. Many of us are not familiar with other package managers, and I think it weakens our brand to always compare ourselves to them. I find the seemingly random references to other package managers to be jarring.

You can
maintain it by hand,

Good

spack:
repos:

• file:///usr/local/spack-repo

Why does repos require a URL but other things in the config just require a filename? That seems a bit confusing. What happens if you give a simple filename for the Spack Repo?

set custom stage areas for the environment

config:
template_dirs:
- /path/to/my/local/module/templates
build_stage:
- /usr/workspace/myusername

Would use of this features limit portability of Spack Environments between different systems? Actually I suppose that any path-based features could... Something should be said somewhere that when you port a Spack Environment, the paths needs to be manually checked / updated. Acutally, there probably needs to be a whole section on porting environments, once we (someone) have gotten practice with it.

The sections in the file can be anything from the regular Spack
configuration files,
so you can do some sophisticated things here if you're creative. You can
also include configs from elsewhere:

spack:
  # include external configuration
  include:
  - ../special-config-directory/
  - ./config-file.yaml

Can this do paths based on env vars, eg:

include:
- {HOME}/my-config.yaml

add package specs to the specs list

specs:

• hdf5
• libelf
• openmpi

Note the change in syntax (YAML grammar) required to accomodate Spack Setup (coming in a future PR on the heels of this one).

https://github.com/citibeth/spack/blob/efischer/giss/var/spack/environments/twoway-dev-gibbs/env.yaml

    specs:
        modele-tests:                          # Highest precedence
            setup: [modele-tests]
        modele:
            setup: [modele, ibmisc, pism, icebin]
        modele-control:
        modele-utils:
        ncview:
        netcdf:
        nco:      

The included items can either be Spack configuration scopes (directories
with packages.yaml, config.yaml, compilers.yaml, etc.) or files
with all of those config sections merged into a single file (like in the
spack.yaml file above).

Wouldn't it be great if this could be the case for the rest of Spack as well and we didn't have to separate our (now numerous) configs into 5 different files each...????

Notice that you can have relative paths. Those are relative to the
spack.yaml file, so you can put it and its associated configuration in
a git repository if you want to.

We should expand more on why this is a good idea. You can fork Spack, then add your own environments to it. Now you have a single turn-key Spack solution for your users. They clone your fork and install the environment(s) you included in it. They don't have to do any crazy setup or configuration. When you want to upgrade your users, you can do it in a controlled manner by updating your environment(s) in the repo, pulling from the latest Spack upstream, etc. And you can test it all before deploying your updated fork to users.

spack.yaml in code repositories

As is common elsewhere in the dependnecy management world (Pipenv, Cargo,
Bundler, etc.), you can put a spack.yaml file in your project's
repository and use it to make bootstrapping dependencies easier for your
users:

$ git clone https://github.com/myproj/myproj.git
$ cd myproj
$ ls
CONTRIBUTING.md  LICENSE  README.md  configure*  m4/  spack.yaml  src/
$ spack install
# ... all dependencies from spack.yaml are installed ...

Woah... I feel we just moved into an entirely new feature space, this is not really Spack Environments anymore. So... it looks like spack install from within a project's git repo installs that project's dependencies? This brings up so many questions:

1. Why don't we get dependencies from a package.py file (or equivalent), which package developers should already be writing, that is included in the repo? Now I'll have to be writing my dependencies so many times: (a) in CMake, (b) in package.py, and now (c) in spack.yaml???
2. Does this, explicitly or implicitly, create, activate, load or otherwise use a Spack Environment?

spack install, when called without arguments in a repo that has a
spack.yaml, will concretize and install all the specs in the
spack.yaml file.

Isn't this normally what happens when you install a package, via package.py? Why do we need a new mechanism for it?

What would make more sense to me is if 3d party repos can ship a combination of Spack Configs and Spack Environments, allowing them to set up a recommended configuration of software versions for their software (as opposed to trying to encode all that in package.py, which has created problems in the past).

In general... I think this feature needs more thought, more documented/expected use cases, etc.

spack.lock and reproducing environments

The spack.lock "lockfile" is created whenever you install or concretize
an environment. If you run spack install as suggested in the previous
section, it will produce a file called spack.lock alongside
spack.yaml. This contains both the abstract specs that were used to
install the packages, and the full, concretized specs of these packages
and their dependencies. It's intended to allow you to reproduce an
environment exactly as it was built by someone else.

If you send someone a spack.yaml or spack.lock file, they can create
a new environment from these files with commands like these:

It seems from the docs this is not a lockfile, so why is it named like one? Is this just the old environment.json file rehashed?

spack env create -e spack.yaml

or:

spack env create -e spack.lock

If you create an environment from spack.yaml

In which directory will that environment be created? Does it copy the spack.yaml file you gave?

, you'll get a new
environment with the same root specs (like a pip requirements file), and

Again... no need to keep referencing other environment managers. It just confuses those of us who are not familiar with them.

If you
create it from spack.lock, you'll get that and the concrete specs to
reproduce things exactly.

Suppose I do:

1. Create spack.yaml. Edit by hand, add comments.
2. Concretize
3. Move spack.lock to a new machine.
4. Create new environment based on spack.lock.

Question: Will comments from (1) be preserved in (4)? If not... is there a procedure that allows for this to happen?

Currently that has to be completely exact,
but in the future we'll support generating something "as close as
possible" to the original environment on a new host.

Huh?

Activating an environment will add paths
for this directory. @scheibelp will be adding that after he reviews
this PR.

It is inconsistent if activating an env with views includes the env into your PATH; but activating an env without views does not. It also brings up the question, what if I make an env and I want to use it sometimes as a view and sometimes by loading a module? Now when I "activate" it, I will always be loading the View directory, even if don't want to be using it that way at this time.

1. There's only one manifest file format -- spack.yaml, and you can use it to initialize a new environment or to control an existing one.
2. spack.yaml is updated with new abstract specs when you use spack env add, spack env remove, etc., and comments are preserved through these updates.
3. spack.lock always contains the results of the last time the environment was concretized (both inputs and outputs). spack.yaml is the human-editable file with inputs, and spack.lock is machine-readable and exact. They're kept in sync.

These are solid improvements.

4. The command line is a bit different, but I hope that the activation concept along with embedding environment commands in regular Spack commands (spack find, spack spec, etc.) solves most of the issues.

I believe this is heading in the right direction. If functionality is missing, it should be possible to add/tweak it in the future, without radically altering the UI.

• Move the old spack env command to spack build-env

I feel we have confusion reigning. We now have the following ways to run something (including maybe a shell) with different env var settings:

• spack build-env
• spack env activate
• Are there any other ways?

Meanwhile, spack activate does something completely different. Let's see if we can clean up the commands here, to avoid eternal confusion.

PS: Does spack build-env work within spack env activate???

I'll just mark the comments on the docs resolved -- the rst docs still need updating, and the PR is supposed to be a high-level version of what they'll look like.

Yes... I would take the "requested changes" section seriously, the rest is commentary.

If you do all your work in Spack Environments, then it is now possible to garbage collect unnecessary packages. This is important on shared supercomputers with inode limits. BUT... Spack has to know about all the environments it is supposed to keep around. Creating environments outside of Spack will break garbage collection (as would using Spack without environments). The idea of creating an environment outside of Spack is an obvious feature; but I would want to have a clear idea of why we need it???

In any case, this downside of this feature should be documented.

There are basically two reasons to do this:

1. It lets people script environments. If I write a workflow that creates/modifies/etc. environments, I have to come up with names for named environments, which is great for UI on my own machine (and easy to remember and manage), but it's a pain for scripting. If I make an "anonymous" environment for a script, now I have to come up with a name for it. We could add the moral equivalent of mktmp for environments, but I think it's more flexible to let the scripter pick the storage location, instead of requiring that everything be managed. I agree we should document that they can't be gc'd (though we don't have a gc yet 😄).
2. You need this feature to support sticking a spack.yaml in a project repo, and I'm trying to leverage what we're doing for environments for UI and developer workflow. The manifest/lockfile pattern is something developers have been asking for, and I don't think any other package manager in the same category in Spack provides it. For example, Conda and Conan both give you the equivalent of requirements.txt but nothing like a lockfile.

FWIW, the managed package thing also comes up with spack chains (#8772). We decided to ignore it there but we plan to have commands that will copy needed packages into a sub-spack, instead of just linking. We could do something similar for environments but I have not thought it through. The model that comes immediately to mind is npm's local vs. global modules.

the rest is commentary

but I like the comments!

Yep I mean spack env activate. I really want to get rid of spack activate once environments are fully fleshed out.

Meaning (2) is not always desired; especially when a Spack Environment is not yet fully built.

I think this can be an option on activate (activate with and without loading the results). I agree you want this for things like x-compiled environments.

Tell them that you've created this nifty module named myenv that they can load with their trusted module load command --- and they will be very happy.

I think it would be easy to also generate modules for environments, with corresponding names, which would provide a familiar interface for HPC people.

@SteVwonder @trws @alfredo-gimenez @bhatele: FYI

@citibeth I think there are several requests you make that can be done later:

Generate a single module for an environment. Remove spack env loads, which always felt like a hack.

Potentially useful but this can be done later

If spack env loads is not removed, make sure that Spack module generation works within an activated environment. It should re-generate modules for just that environment.

spack env loads only loads modules for specs in the environment. It doesn't affect module generation (or regeneration). If there is module-specific configuration, then it occurs to me that the module files for different environments will all be merged into the same directory (with potentially different schemes). IMO this can also be handled later.

Integrate Spack Environment garbage collection

This should happen after this PR is merged. GC is a great feature but not essential for core environment functionality.

Comments on UI consistency:

Once a user has run spack env activate , they are still required to (unnecessarily) give their env name again when they wish to modify it (eg: spack env add)

I don't think this is true: if an environment is active, spack env add <spec> will add the spec to the current active environment (see cmd/env.get_env).

That being said, there is the possibility of inconsistency for spack env add: the spack env add command has a -e option, so there are 3 possible specifications of the environment: spack -e <env1> env add -e <env2> spec, with the third environment being the currently active one (i.e. the SPACK_ENV variable is set).

To make this consistent with other commands, the spack env add -e functionality could be removed.

@tgamblin on that count, why was -e added to Spack's main.py vs. managed as a common arg (added to cmd/common/arguments.py)? I'm guessing it has something to do with config initialization but it would be helpful if you stated it here.

Some operations on a single environment exist as spack env sub-commands (eg: spack env add), whereas others exist as top-level commands (eg: spack install). Still others are top-level commands that take the environment name as an ad-hoc -e flag (eg: spack cd).

I think that in different cases each of these approaches makes sense. This depends on whether:

• a given action only makes sense within an environment
• an existing Spack command could be modified to operate in the context of an environment

spack env add only makes sense in the context of an environment. One could offer spack add as a shorthand but IMO that would be confusing: exposing environment-specific operations as subcommands of environments is more helpful; I think having two top level commands spack add and spack install would be confusing.

spack -e <env> install vs. spack env install vs. spack env install <env name>: spack install exists separate from environments and intends to install a single spec; it can take on additional meaning in the context of an active environment. spack env install installs all specs in an environment and depends on activation of the environment; spack env install <env name> does the same without requiring activation of the environment. Of the 3, only the 3rd is redundant (with the 2nd), but it is also the one that is the least important and the easiest to control.

Regarding spack cd: Other than spack cd, all Spack commands customize their environment by setting spack -e <env> command ... rather than spack command -e <env> .... I think spack cd is a special case. This is already something of a catch-all command: it lets you relocate your CWD to various locations that are relevant to Spack (it is based on spack location). Unlike spack install it isn't necessarily related to the current environment. For example spack cd -P takes you to the package repository directory.

EDIT: This is an awesome feature! Thanks to everyone that helped bring it to life.

It seems from the docs this is not a lockfile, so why is it named like one? Is this just the old environment.json file rehashed?

I don't know the origin of the lock, but I know Gem and NPM use "lock files" to store the concrete package versions installed. I agree that the naming is confusing, and it might be best to break with convention here and name it something else. Like maybe spack.concrete?

Consider renaming spack find to spack status, since it's always been highly confusable with spack list.

I agree that the list vs find naming is very confusing, but based on the open issue (#4159) I don't think there is consensus on what to change them to. I would recommend against changing the names of any top-level commands in this PR, as that would be a large breaking change. Maybe spack env status could become spack env find if consistency is of utmost importance.

Now... is it possible to use a single YAML file for a config, rather than separate packages.yaml, etc? That would be nice.

Wouldn't it be great if this could be the case for the rest of Spack as well and we didn't have to separate our (now numerous) configs into 5 different files each...????

👍 👍

I really want to get rid of spack activate once environments are fully fleshed out.

👍 👍

(The stuff below is largely an echo of @scheibelp's comment above)

Integrate Spack Environment garbage collection: scheibelp#1

I wouldn't hold up this PR waiting on this feature to be implemented. I think this should be a follow-on PR.

Move spack env add SPEC to the top-level command spack add SPEC. Same for spack env remove. It's an error if these are run without an environment.

Move spack env concretize to the top-level command spack concretize. It's an error if it's run without an env.

Not that I have any great alternatives in mind, but this smells off to me. It seems weird to have top-level commands that only work inside an environment. These commands seem ripe for sticking under spack env.

More specifically, I have an opposition to adding spack add as a top-level command. Both spack add and spack install, which semantically are very similar, would be top-level commands. I could see having both as top-level commands being confusing to new users, just as list/find and clean/purge can be confusing.

Sort of related: does anyone have a spack "cheat sheet" of commands? Having them all laid out in one place and organized by topic could be useful for this kind of discussion.

I wanted to add a note on "lockfiles". The manifest/lockfile model has already become a thing -- it refers to locking the resolved versions of dependencies in most of the language-specific package managers out there right now. Here are some references:

• Ruby: Bundler has Gemfile and Gemfile.lock
• Java: Gradle has classpath lockfiles
• Rust: Cargo, which I believe traces its lineage to Bundler, has cargo.toml and cargo.lock
• Javascript: npm has package.json and package-lock.json
• Javascript: Yarn has package.json and yarn.lock
• Go: dep has Gopkg.toml and Gopkg.lock
• Go: Glide has glide.yaml and glide.lock
• Python: Pipenv has Pipfile.toml and Pipfile.lock
• Dart: Pub has pubspec.yaml and pubspec.lock
• PHP: Composer has composer.json and composer.lock
• Chapel: Mason has Mason.toml and Mason.lock

Honorable mentions:

• C++: Conan doesn't have lockfiles, but it has conanfile.py and people have asked specifically for a .lock mechanism
• Homebrew: homebrew-bundle's README mentions that brew doesn't support lockfiles because brew doesn't have flexible ways to set specific versions.

I went with spack.yaml and spack.lock because I see this model growing in the project dependency management space (all these package managers are in that space except Homebrew, and brew-bundle attempts to provide this for Homebrew). People have come to expect it, and I don't think we should try to rename something that has become a very familiar concept.

I do think we did better than these package managers by having notions of "abstract" and "concrete" specs and "concretization". I think that's a clearer concept than "dependency resolution", which is what most of the other tools call it. Dependency resolution typically only refers to setting specific versions, while concretization does that and compilers, variants, flags, etc., and I think that's an important distinction. I don't think it's hard or unintuitive to say "the spack.lock file contains all the concrete specs for an environment."

@tgamblin I think the most important outcome so far on the UI discussion is that the extra -e option should be removed from all spack env subcommands (i.e. no more spack env add -e), since it is already specified as part of spack -e . That will avoid confusion WRT redundant environment specifications and keep things mostly consistent (except for spack cd). Relatively speaking, the rest of this is less important.

(less important) I argue below that spack -e <env> <command> ... should be spack <command> -e <env> ... (note this does not conflict with the request to get rid of spack env add -e <env> above - I am saying spack -e <env> env add should be spack env -e <env> add) but that's relatively speaking not as big a deal.

@citibeth

Because spack -e FOO command... is the same as spack env activate FOO; spack command. You can do the latter for any command, even when the command itself is insensitive to environemnt (eg spack list). So there's really no harm in being able to set the environment for any command using spack -e, even if the command doesn't use it.

You can already activate the environment-aware version of any spack command by first doing spack env activate <env> and in that case you do even less typing. If you change spack -e <env> command ... with spack command -e <env> ... you then get a clear account of which commands are environment-sensitive "for free" (i.e. built into the command help) vs. having to read through docs. This would avoid the confusion with spack cd.

You forgot spack -e <env1> install <env2>. Which would be an error?

I didn't. That command is not valid. I do think though that users may end up getting confused about this unless one of the environment specification mechanisms is removed.

That being said, there is the possibility of inconsistency for spack env add: the spack env add command has a -e option,

A few points here:

1. You have already put a global -e option on the whole Spack command.

The argument you are quoting in #9612 (comment) agrees with your response, which appears to be phrased as though it is a counterargument, so I am confused.

@scheibelp @becker33: I integrated the setup-env.csh fix and removed docs for now to add them in a PR very soon.

We are still having an issue where Python 2.6 builds are hanging and we don't know why. We've seen this in some other builds as well. If it persists on develop we can let Python2.6 fail until we fix it; the Python 2.6 tests pass when I run them and when I log into a Travis container and run them, so I believe the failures are related to Travis, not Spack.

At any rate, this is merged! Environments are in! Docs, a tutorial, and some new feature additions are coming soon.

@tgamblin are there any plans to marry a repo's spack.yaml with a simplified package.py?

That would allow software developers to in-source describe and update dependencies instead of writing a lot of "if between release X and Y, depends on [email protected]:2.5, else [email protected] and if Y>3.4 also adds dependency on W" inside a single package.py. They get cluttered over time quickly. Maybe I am also missing some of the possible workflows.

From the timestap, it looks like some midnight oil was burned on Spack Environments. Thank you Peter and Todd for all your hard work. This is truly a glorious day in Spack Land!

That would allow software developers to in-source describe and update dependencies instead of writing a lot of "if between release X and Y, depends on [email protected]:2.5, else [email protected] and if Y>3.4 also adds dependency on W" inside a single package.py. They get cluttered over time quickly. Maybe I am also missing some of the possible workflows.

@ax3l: I think there's a long-term path there but right now that is hard. We're not a distributed package management system (yet) and the HPC ecosystem needs some packages to be curated. There is probably a balance where some packages can be managed more like they are in registry-based systems, where you have a package.py vary over time in a repo, and others can just be central ones. But I wouldn't expect that soon. I do like the idea, though.

+1 I tend to agree with the need for curation in the various environments or stacks as a starting guide for others to modify locally