Skip to content

Spack Environments 3.0: How they should REALLY work #7944

@citibeth

Description

@citibeth

@scheibelp @tgamblin
The ideas in this Issue might get implemented, but only after the current Environnments PR (#7843) is merged.

Rather than a spack env command that increasingly duplicates much of Spack, Environments should be integrated into existing Spack commands. The following commands could be "environmentized": find, all "build packages" commands, all "environment" commands (in spack help --all), module.

Commands would be run with an environment in maybe one of a few ways:

  1. spack --env <env> command ... --- works for all commands, but only makes a difference for commands where it makes sense.
  2. SPACK_ENV=<env> spack command ... --- get the current Spack env out of an env var, but otherwise similar to (1). Nothing else is done in Spack this way, so probably not what we want.
  3. spack command --env <env> --- Add the --env flag only for commands where an environment makes a difference.
  4. Don't use any global env specifier; instead, get the required env from the specs on the command line (see below).

Suppose we have an Environment with user specs aa, bb and cc added to it. Then the following conventions (or something like them) can be used anywhere a spec is asked for on the command line:

  1. env:aa refers to the aa user spec in the current environment. If aa has been concretized, then it will be replaced with the concretized version of aa in the env. NOTE: This is different from just aa, which does an immediate concretization.
  2. env: expands to all user specs in env.
  3. env:aa: expands to all specs in the aa DAG.
  4. env:aa:xx refers to a spec matching xx, which is in the DAG of the concretized aa user spec. This kind of control allows us to easily debug, rebuild, etc. individual problem packages within an environment.
  5. env::xx refers to a spec matching xx anywhere in any DAG in the concretized environment.
  6. env:: expands to ALL SPECS in the environment.

IN general:

  1. Anything wanting an environment has no colons. Eg: myenv
  2. Anything wanting a user spec, or list of user specs, or top spec in a DAG, has one colon. Eg: myenv:aa or myenv:
  3. Anything wanting any spec has two colons. Eg: myenv:aa:openblas.

spack install would work as it currently does: it concretizes and installs all in one. spack install w.r.t an env would add the user spec to the env, concretize it and then build ("install") it. For example, spack install c1:emacs would add emacs to the c1 environment, concretize it, and then install.

spack add adds user specs to an environment. Eg: spack add c1:emacs

spack concretize concretizes one or more user specs. Eg: spack concretize c1: or spack concretize c1:aa

spack build builds ("install") concretized specs in an environment (concretizing them if not already). Eg: spack build myenv:, spack build myenv:aa, spack build myenv:aa:openblas.

Thus, spack install is equal to spack add + spack concretize + spack build.

spack location, spack find, spack load, spack module loads can now be used to find installed packages in a deterministic manner. (Thus, they are no longer "broken" as a reliable way to obtain hashes of stuff you've built and now want to use). Eg:

spack load -r myenv:aa   # Loads all modules in myenv:aa DAG
spack location -i myenv::openblas   # Finds the (hopefully 1) openblass in myenv
spack module loads -r myenv:   # List all modules needed to load myenv

Note that spack activate should not be used with environments. You need to either use spack module loads, or spack view.

Thoughts?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions