Closes #15939
Closes #49939
Closes #11871

This is a redo of #11871, which aims to make Spack pip/apt-installable. Use cases are described in detail here (they are numbered, and this PR description may refer to them like "U1"), along with parts of 11871 that are not handled here.

Moving Spack-generated artifacts outside of $spack

The biggest blocker to creating a shared spack instance is that Spack will default to writing data into its own prefix. For new clones of Spack, this PR moves all generated files out of the cloned directory and into ~. This includes:

1. installs
2. where environments are stored
3. cached downloads
4. gpg keys
5. where modules are generated

For old clones:

• If no new spack clones are generated and only old spack clones exist, then each of them will continue using their old install location inside of their respective $spack directories.
• If a new clone starts writing into ~ and an old spack instance git pulls this logic, the old instance will start using ~. A warning is generated in this case

1-3 are large. Before this PR there were config options to relocate each of these individually (and those still take precedence). This PR offers additional mechanisms to set a single env var or config option in order to relocate all of them together.

Controlling install location

This PR establishes the following order:

1. config:install_tree:root:...
2. SPACK_DATA_HOME env var if that is set
3. SPACK_HOME env var if that is set
4. config:locations:data if that is set
5. config:locations:home if that is set
6. XDG_DATA_HOME env var if that is set
7. Under the default for XDG_DATA_HOME: ~/.local/share/spack

If an older clone of spack pulls this, and has existing installs in $spack/opt/spack (the old default), then:

• [1-5] always take precedence
• [6,7] take precedence if they are not empty

Other notes:

• This PR maintains a default setting for config:install_tree:root so that users who do spack config get config will see that setting, but it uses a special default variable that effectively behaves as if it were unset.
• This applies similarly for envs, but the associated config vars is config:environments_root
• Downloads are relocated unconditionally (although are still overridden by config:source_cache)
• All of 1-3 from the prior section can be relocated by setting SPACK_DATA_HOME, XDG_DATA_HOME or config:locations:data
• Everything (old and new) that would be written to $spack or ~ before can be relocated by setting SPACK_HOME or config:locations:home, except for the user config scope
  • Update Feb 11 2026: etc/spack/include.yaml sets path: "$spack_home/.config/spack/" for the user scope. In order to influence this, the user must set the SPACK_HOME env var, or they must set config:locations:home at a scope that is lower priority than the spack scope (right now that is awkward - see: Shared spack #47615 (comment))
  • Update Mar 5, 2026: undoing change from prior point: allowing include: config to reference config-based location variables causes too many headaches
• The user can disable env vars by setting config:locations:disable_env:true (i.e. 2, 3, 6 will be ignored).

Admin management, pip-installed spack

• When this PR was first made, new installations would prefer to install in $spack. As of now, everything goes into ~. I think this default behavior will be sufficient to support pip installed spack
• If admins want to provide an upstream to users but avoid using it themselves, they can use existing include: config in the spack scope:
  • Include a scope that adds upstreams:x config when: SPACK_ADMIN not in env
  • Include a scope that adds install_tree:root:x config when: SPACK_ADMIN in env
  • As long as all admins make sure to define this env var, this will satisfy [5a/b], and

avoiding ~

Before this PR, you could do

export SPACK_USER_CACHE_PATH=x
export SPACK_DISABLE_LOCAL_CONFIG=1

and nothing would be read from or written to ~

With this PR, that would not be enough. Some ways to do that here:

• This PR includes a new spack isolate command that you can run to configure spack to write into $spack
• If you set SPACK_HOME=x SPACK_USER_CONFIG_PATH=x/config, everything will be written to x
• Or you can do SPACK_HOME=x SPACK_DISABLE_LOCAL_CONFIG=1
• If you set SPACK_DATA_HOME or config:locations:data outside of ~, this will relocate everything that takes up significant space (envs, installs, cached downloads)

To completely move everything, in an env you can do:

include:: []
config:
  locations:
    home: x

Users who have write access to $spack can do

spack config --scope=spack edit include
spack config --scope=spack add config:locations:home:x

XDG compliance

Everything that was written into ~/.spack is now relocated to XDG-compliant alternatives:

• Default user config scope location (now in ~/.config/spack)
• Default user cache path location (which includes the misc cache) now in ~/.local/state/spack
  • Overridden by $SPACK_USER_CACHE_PATH (highest priority), $SPACK_STATE_HOME, and $XDG_STATE_HOME

Everything moving out of $spack is moving into XDG-compliant default locations under ~.

• (4) gpg keys move to $SPACK_DATA_HOME/gpg-keys
• (5) Default module location moved to $SPACK_DATA_HOME (e.g. $SPACK_DATA_HOME/lmod)

See also: #49939

Crosstalk between spack instances

• The misc. cache now includes a per-spack component (based on a hash of the Spack root)
  • config:misc_cache still exists, so users can share a single misc cache among multiple instances of spack
• Ideally each individual spack would have a distinct highest-level write scope, but for now they all share ~/.config/spack

Use cases/considerations

All items here are addressed, although there is some friction with [4]

1. apt install of Spack
2. pip install of Spack
   a. sub-case, with --user (i.e. user may have write permissions to the install location)
3. Admin provides spack to users
   a. but users might choose to clone spack themselves
   b. and they may or may not want to use an admin-curated upstream: it should be possible for users to get an upstream DB by default from an admin without running any extra commands
4. ~ may or may not have significant storage space
5. simple config management
   a. admin should be able to define config that only applies to users, not admins
   b. Multiple admins may all want to share a config, but not provide that config to an end-user
   c. users may want their install location to differ on a per-spack basis (but still live outside of the spack prefix)
6. Handling new/old spack instances: the logic should make sense for...
   a. new git clones of spack
   b. git pulls in existing clones (and e.g. should be able to use gpg keys)
   c. Discarding a pre-existing download cache is not a problem

TODOs

• Testing

Maybe TODOs

• Admins may want to provide gpg keys to users, but right now that is not configurable
• It could make sense to provide a command that relocates gpg keys, cached downloads, modules, and environments from their "old" locations (inside of Spack) to their "new" locations
• In the case of a pip install, this is only intended to be useable as a cmd-line tool and not importable: nevertheless, it may be useful to strip out vendored libs (i.e. code in lib/spack/external/) and handle it w/requirements.txt.
  • That is possibly complicated if the versions we vendor (intended to support back to Python 3.6) are no longer supported/available
  • UPDATE: since this PR came out, the top-level vendored modules moved into a spack.vendor namespace, so long as those top-level modules do not themselves require vendored libs, this would prevent vendored Spack python deps from interfering with imports.

Things not translated from #11871

If prefixed w/ "drop", I think we can skip it entirely

• (todo, possibly in another PR) Upstreams were refactored to work without configuration
  • This is considered a good idea, but isn't strictly required to make Spack apt/pip-installable
• (todo, possibly in another PR) 11871 moved module roots to the install tree root. This another improvement for upstreams, although not strictly required to support admin/end-user use cases
• (drop) 11871 had a notion of managing multiple install trees and being able to select them with e.g. spack --install-tree=... install
  • I claim this implementation covers the same use cases without depending on this concept. For example there was a notion of an end-user install tree and an admin install tree, and user-install trees were automatically directed to the admin install tree as an upstream
• (drop) 11871 introduced a configuration scope for each installation root. Roughly speaking, this is awkward because store.py depends on config (so this would introduce a cyclic dependency).
  • (drop) Permissions customization for install trees: with per-install-tree config, it became useful to distinguish desired permissions for the admin install tree vs. the end-user install tree: this use case is now addressed with admin-specific config scopes

Other changes

Not essential to this PR, but related to other changes made here

• This has a rudimentary audit to detect writes into the Spack prefix (sys.addaudithook); it is enabled with --guard-writes-into-spack