Skip to content

[WIP] Chaining shared Spack#15939

Open
scheibelp wants to merge 28 commits intospack:developfrom
scheibelp:features/chaining-shared-spack
Open

[WIP] Chaining shared Spack#15939
scheibelp wants to merge 28 commits intospack:developfrom
scheibelp:features/chaining-shared-spack

Conversation

@scheibelp
Copy link
Copy Markdown
Member

@scheibelp scheibelp commented Apr 8, 2020 •
edited
Loading

(This is very preliminary and I don't recommend even trying to use this at this point, but it covers some of the most essential changes that need to be made)

Changes which allow for a single system-installed Spack, with an installation root maintained by a system admin and an installation root maintained by a user (but for example admins can set config values that apply to any user of the Spack instance). Overall this is intended to allow admins to deploy an instance of Spack which looks like any other system-installed tool.

This includes:

  • Move module roots to install tree root (not strictly required but in general for users they need to be located outside the Spack prefix)
  • Add config scope associated with the install tree itself (this allows admins to assign specific permissions to the packages installed that are meant to be available to all users)
  • (5/20/20) Environments are stored in the install root. This allows users with a shared Spack instance to make their own environments. This does not allow admins of the shared Spack instance to provide environments (that could be handled in a later PR)
  • (5/20/20) For now it is assumed that users would not need to manage their own GPG keys: they would use the admin-created/added keys to sign new binary caches or to install them.

TODOs:

  • Testing
  • Move download cache to ~ (users can't store cached archives in the Spack prefix)

Non-essential TODOs:

  • It came up that indices (e.g. the provider cache) should be instanced on a per-install-tree basis, that is important for correctness in general but is not essential to this PR
  • If users of shared Spack could access the download cache used by the admins, this would reduce downloads and space used across all users of the shared Spack instance. For now, admins can create mirrors and add them to the config that users see.

See also: #11871

scheibelp added 18 commits March 30, 2020 18:13
@scheibelp
allow reading upstream location from install root (this does not yet …
10b09b1 
…store the install location in the install root)
@scheibelp
add cmd-line options to specify install tree and shared upstream; add…
870323b 
… config for recording available install trees and shared install trees; add logic for initializing spack install location based on provided options and config
@scheibelp
fix typos + remove unneeded arg
9cd06a6
@scheibelp
reference error
b8e32db
@scheibelp
intermediate work on bootstrapping install tree configuration into co…
5c5963a 
…nfiguration scope assembly
@scheibelp
handle config initialization when config is not set for install_trees…
3f63005 
…/shared_install_trees
@scheibelp
remove redundancy and add clarifying comment
5686508
@scheibelp
fix spack default install prefix
5cb2843
@scheibelp
adjust scope precedence
a8d9930
@scheibelp
remove 'install_tree' from config: spack doesn't use it anymore. Note…
c5d9396 
… that container logic still expects it to exists, so that needs an update
@scheibelp
add comment for special error case
f7d5854
@scheibelp
intermediate work on moving upstream modules to install root
9a17520
@scheibelp
read module roots assuming that modules are stored in install tree
f3f55f3
@scheibelp
actually return value
c24acc6
@scheibelp
resolve lazy reference
4b062bf
@scheibelp
generate modules in the install tree by default; automatically look f…
18568df 
…or module indices in upstream install trees (in particular if module paths are not configured for upstreams)
@scheibelp
clarifying comments; handle user-specified install tree being one of …
dae550a 
…install_trees vs. shared_install_trees
@scheibelp
remove redundant code
138398d
@tgamblin tgamblin self-requested a review April 8, 2020 21:38
@scheibelp
Copy link
Copy Markdown
Member Author

scheibelp commented Apr 14, 2020

Add config scope associated with the install tree itself (this allows admins to assign specific permissions to the packages installed that are meant to be available to all users)

In more detail:

  • There is a config scope associated with the install tree
  • If the user runs spack --install-tree=X and X appears in config:shared_install_trees, the user config scope (i.e. from ~/.spack) is ignored and the installation-tree config scope is loaded
  • If the user runs spack --install-tree=Y and Y is not a shared install tree (i.e. not in config:shared_install_trees), then the user config scope is loaded along with the installation tree config scope (in that order, with values from the install tree overriding values from ~)
    • Because of this ordering, the user can configure install trees in ~

@tgamblin tgamblin mentioned this pull request Apr 28, 2020
Merged
@scheibelp scheibelp mentioned this pull request May 29, 2020
Closed
8 tasks
@scheibelp scheibelp mentioned this pull request Jan 22, 2021
Open
@alecbcs
Copy link
Copy Markdown
Member

alecbcs commented Sep 27, 2024

@scheibelp is this still something you're interested in merging or has this since been fixed elsewhere?

@alecbcs alecbcs added the stale label Sep 27, 2024
@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Sep 27, 2024

@alecbcs we are going to get this or something like it in at some point, but have not had the cycles to pick it back up.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Nov 26, 2024 •
edited
Loading

@tgamblin Can we close this since it is superseded by #47615 ?

@scheibelp scheibelp mentioned this pull request Nov 26, 2024
Open
1 task
@scheibelp
Copy link
Copy Markdown
Member Author

scheibelp commented Nov 26, 2024 •
edited
Loading

@alalazo for now I've added closes #this-pr to #47615

(I prefer doing that in case the PR that was supposed to replace this gets closed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tgamblin tgamblin Awaiting requested review from tgamblin

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@scheibelp @alecbcs @tgamblin @alalazo @haampie @adamjstewart