What is the expected timescale of merging this PR? I haven't looked into this PR too deeply yet, but if this can be made compatible with filesystemviews our setup and deployment chain here would get much simpler and more robust.

What is the expected timescale of merging this PR?

@healther this PR is ready and integration with filesystem views is expected in January

what is the timescale on merging this? I'd really like to start using this

@healther: Latest September but this will probably be merged much much earlier.

Peter, this is really nice! Sorry it took me a year to get around to looking at it; but in that year, I had everything I needed already built, and I didn't need to really use Spack. I was pleasantly surprised to see that even config scopes had made it into the current version.

Here are issues I've found on an initial trial of the system:

TODO: Necessary Features

There are features that I consider necessary for my work.

• Need to integrate Spack environments with spack setup. The command spack env add should be upgraded to allow for certain packages in the DAG being added to be setup instead of installed. For example: spack env add c1 bigpackage --setup bigpackage,mypackage. When spack env install c1 is called later, this will cause the files config-bigpackage.py and config-mypackage.py to be written out (rather than installing those two packages). The user is then required to use the config-xyz.py files to build and install manually with CMake (or whatever build tool).

Functionality Issues (Minor)

• spack env commands should indicate where the environment is stored. spack env create and spack env list. The only way I know how to do this so far is with spack env add fake-environment fake-spec (and spack env concretize), and inspecting the error message.

Bugs

• spack env list-modules c1 produced the error message:
  ==> Error: 'core_compilers' key not found in configuration file. Probably

• The command spack env create ./c1 should create an environment named c1 in the current directory. Instead, it create spack/var/spack/environments/c1. In general, the use of environments outside the Spack tree seems to be buggy.

Documentation Issues

• How can I see the fully concretized spec for a particular spec in an environment? spack env list xyz only shows the top-level of the DAG. Also needs a way to show the fully concretized spec of all specs in an environment; along with which specs have

• What is the difference between spack env concretize and spack env relocate? Why is spack env relocate not called spack env reconcretize?

• Every spack env command needs to be documented in environments.rst. Currently missing are remove, relocate, list, upgrade, stage, update-config. Working examples in the docs, not just descriptions, would be super!

• Documentation is needed on the creation and use of Spack environments stored inside vs. outside of spack/var/spack/environments.

• Can an environment contain multiple specs that conflict? If so, this needs to be documented, along with examples of how it can work out / be useful in the end (i.e. how it relates with modules).

TODO: Nice-to-Have Features

These features would be nice, but I can do my work even without them.

• Multi-Modules: Instead of using spack env list-modules to create a list of modules to load, it would be nice if spack env could create a single module equivalent to all the modules in spack env list-modules. Then the entire environment could be loaded / unloaded with a single module load command.

• Referenced config scopes: Instead of containing a config scope, an environment could point to one or more config scopes sitting elsewhere on the filesystem. For example, I could create a config scope fedora.config/, containing some basic stuff only relevant to Fedora. When setting up my Spack environment on Fedora, I have it use that config scope (in addition to its own). If I later move the environment to an Ubuntu system, I can have it point to ubuntu.config/.

(this doesn't address all of the comments at #4585 (comment), e.g. the part about spack setup - still thinking about that)

How can I see the fully concretized spec for a particular spec in an environment? spack env list xyz only shows the top-level of the DAG.

spack env list --include-deps

What is the difference between spack env concretize and spack env relocate? Why is spack env relocate not called spack env reconcretize?

spack env relocate attempts to preserve all features of the spec dag other than the operating system and compiler, while spack env concretize is used to concretize whatever specs the user added. A command like spack env reconcretize (not currently present) would start again with just the specs as provided by the user.

Can an environment contain multiple specs that conflict?

A user cannot explicitly add the same top-level package twice. A package may appear in the environment multiple times as a dependency (or once as a top-level package and multiple times as a dependency). In this case when it comes to listing out associated modules, for example, it will report only the first instance of the package (in terms of the order that the user added the specs to the environment).

The command spack env create ./c1 should create an environment named c1 in the current directory. Instead, it create spack/var/spack/environments/c1

To be clear, you consider this essential? I'd prefer to add this functionality in a follow-up PR.

One suggestion... consider changing the command format from spack env <subcommend> <envname> to spack env <envname> <subcommand>. This makes it a LOT easier to type and edit various Spack commands in Bash.

I have copied the branch features/context from the scheibelp fork to the main Spack repo on GitHub. That makes it easier for me to add my own branch on top of this work: features/context_setup, which adds spack setup functionality both as a stand-alone part of spack install, and also as a features of Spack Environments.

I would prefer a little bit of refactoring on this PR:

• Methods on Environment are inconsistent. Some are class methods, some are external functions.

• Methods like environment_xyz(args) are not very useful. Why can't the user just type environment_xyz(args.environment)?

• Better separation between functionality and UI would be nice. There should be an environment API that can be called apart from things like args.

• It's better not to rename packages without a GOOD reason. sjson and syaml should be replaced with the standard names spack_json and spack_yaml.

That said, I think it's best to get this merged first before cleaning up code in this way.

I'm having a problem with this PR just creating and installing a simple environment. Am I doing something wrong?

$ bin/spack --debug env create c2
$ bin/spack --debug env add c2 emacs
$ bin/spack --debug env concretize c2
==> Check for config scope at /Users/rpfische/spack/scratch/var/spack/environments/c2/config
==> Using config scope at /Users/rpfische/spack/scratch/var/spack/environments/c2/config
==> Reading config file /Users/rpfische/spack/scratch/etc/spack/defaults/packages.yaml
==> Reading config file /Users/rpfische/spack/scratch/etc/spack/defaults/darwin/packages.yaml
==> Reading config file /Users/rpfische/.spack/packages.yaml
==> READ LOCK: /Users/rpfische/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> READ LOCK: /Users/rpfische/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquired]
==> READ LOCK: /Users/rpfische/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> WRITE LOCK: /Users/rpfische/.spack/cache/providers/.c2-index.yaml.lock[0:0] [Acquiring]
==> WRITE LOCK: /Users/rpfische/.spack/cache/providers/.c2-index.yaml.lock[0:0] [Acquired]
==> WRITE LOCK: /Users/rpfische/.spack/cache/providers/.c2-index.yaml.lock[0:0] [Released]
==> Reading config file /Users/rpfische/.spack/darwin/compilers.yaml
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/emacs/package.py to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/emacs
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/ncurses/sed_pgi.patch to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/ncurses
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/ncurses/patch_gcc_5.txt to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/ncurses
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/ncurses/package.py to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/ncurses
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/pkgconf/package.py to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/pkgconf
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/zlib/w_patch.patch to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/zlib
==> Installing /Users/rpfische/spack/scratch/var/spack/repos/builtin/packages/zlib/package.py to /var/folders/n2/0ll2lwm50b91f5ppmyggd_g4cdc3y3/T/tmp0xamo1/repo/packages/zlib
$ bin/spack --debug env install c2
Traceback (most recent call last):
  File "bin/spack", line 85, in <module>
    sys.exit(spack.main.main())
  File "/Users/rpfische/spack/scratch/lib/spack/spack/main.py", line 577, in main
    return _main(command, parser, args, unknown)
  File "/Users/rpfische/spack/scratch/lib/spack/spack/main.py", line 469, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/Users/rpfische/spack/scratch/lib/spack/spack/main.py", line 381, in _invoke_spack_command
    return_val = command(parser, args)
  File "/Users/rpfische/spack/scratch/lib/spack/spack/cmd/env.py", line 617, in env
    action[args.environment_command](args)
  File "/Users/rpfische/spack/scratch/lib/spack/spack/cmd/env.py", line 439, in environment_install
    environment.install()
  File "/Users/rpfische/spack/scratch/lib/spack/spack/cmd/env.py", line 107, in install
    spec.package.do_install()
  File "/Users/rpfische/spack/scratch/lib/spack/spack/package.py", line 1390, in do_install
    % self.spec.name)
ValueError: Can only install concrete packages: emacs.

(I believe I have fixed this bug; see #7843)

Continued at #7843