Skip to content

WIP, ENH: remove ~/.spack requirement#13139

Closed
tylerjereddy wants to merge 2 commits intospack:developfrom
tylerjereddy:issue_12892_spack_home
Closed

WIP, ENH: remove ~/.spack requirement#13139
tylerjereddy wants to merge 2 commits intospack:developfrom
tylerjereddy:issue_12892_spack_home

Conversation

@tylerjereddy
Copy link
Copy Markdown
Contributor

@tylerjereddy tylerjereddy commented Oct 10, 2019 •
edited by alalazo
Loading

  • it is no longer absolutely required that
    ~/.spack be used as the user-level config
    scope path---it can now be customized
    to any arbitrary system path

  • ~/.spack remains the default for backward
    compatibility, but a custom value may
    be provided in etc/spack/defaults/config.yaml

  • to accomplish this, it was necessary to
    generate a second Configuration instance containing
    only the default scope, so that this scope
    could be parsed for the user_path setting
    when populating the true Configuration
    scope

Fixes #12892
Fixes #13057
Closes #7433

cc @junghans @AndrewGaspar @alalazo

Notes

  • the diff makes it look more complicated than it really is because of the shifting around of functions in the config scope code
  • if the core team thinks this might work, I could use some guidance on unit tests---for now, I've been able to change the new user_path variable value and my repos and mirror lists get cleared, and if I add new ones then the new user_path starts getting populated with config files, which is a good sign, I think

@tylerjereddy tylerjereddy force-pushed the issue_12892_spack_home branch from cc3e8e8 to 3aa9180 Compare October 10, 2019 23:27
tylerjereddy
tylerjereddy commented Oct 10, 2019
View reviewed changes
becker33
becker33 requested changes Oct 16, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@becker33 becker33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty strongly opposed to a usage model that suggests users should change $spack/etc/spack/defaults/*. Those files should be read-only. Would the same thing be possible with the system scope $spack/etc/spack/config.yaml

@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Oct 18, 2019

I've pushed some revisions, including a draft regression test, but I still need to work on this/test more. I think something is still not quite right with the scoping for example, when used outside the unit test context.

@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Oct 18, 2019

Would the same thing be possible with the system scope $spack/etc/spack/config.yaml

It looks like any scope should be possible, except the user scope itself of course. That said, switching to the system scope didn't quite work in my hands locally when I played with it. Well, spack config get config got the user_path value from the system scope correctly, but it was still polluting the previous / default user_path when I added a mirror (in contrast to the unit test).

I'll have to work on that.

@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Oct 18, 2019

Ah, I figured that out, I do have to push_scope for each scope that is allowed to provide user_path. That's what the unit test does too.

I'll work on more cleanups before I push again though.

@tylerjereddy tylerjereddy force-pushed the issue_12892_spack_home branch from 62d20c3 to 216a1cc Compare October 18, 2019 20:18
@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Oct 18, 2019

Alright, I think I've addressed reviewer comments so far. Tests are passing locally & I added another test in there too, for cache handling with custom user_path.

Added some docs.

tylerjereddy
tylerjereddy commented Oct 18, 2019
View reviewed changes
lib/spack/docs/config_yaml.rst
that does not point to ``~/.spack/stage``.

Setting ``user_path`` at configuration scopes with higher precedence
than ``site`` is not currently supported.
Copy link
Copy Markdown
Contributor Author

@tylerjereddy tylerjereddy Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If core devs think this would be useful, I could look into it, but I'll hold off on that time investment for now. Getting the regression tests reasonably appropriate has been the most time consuming thing.

@tylerjereddy
ENH: remove ~/.spack requirement
01b6171 
* it is no longer absolutely required that
~/.spack be used as the user-level config
scope path---it can now be customized
to any arbitrary system path

* ~/.spack remains the default for backward
compatibility, but a custom value may
be provided in etc/spack/defaults/config.yaml

* to accomplish this, it was necessary to
generate a second Configuration instance containing
only the default scope, so that this scope
could be parsed for the user_path setting
when populating the true Configuration
scope
@tylerjereddy
MAINT: PR 13139 revisions
b14c214 
* added a unit test, test_get_config_user_path_priority(),
to verify the configuration scope priority hierarchy
for setting user_path & to ensure that new configuration
files are placed in the highest-priority user_path without
polluting the lower-priority user_path entries

* add some clarifying comments to config.py with regard
to configuration scoping of user_path; it was also
necessary to individually push each scope that will
be allowed to provide user_path in the early fetch
phase

* add a second unit test to ensure that misc_cache
gets re-rooted to custom user_path, when provided;
a minor library code adjustment was needed to allow
a change in user_path to dictate the change in misc_cache

* add documentation for `user_path`

* remove a spurious os.path.join() in config:user_path
cache path resolution

* adjust the comment language describing removal of
user_config_path such that it directs setting user_path
at the system scope or a higher priority scope
@tylerjereddy tylerjereddy force-pushed the issue_12892_spack_home branch from 216a1cc to b14c214 Compare October 19, 2019 23:30
@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Oct 19, 2019

Updated with fix for Python 2.7/3.5 in one of the new unit tests.

@tylerjereddy tylerjereddy mentioned this pull request Oct 25, 2019
Closed
@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Nov 5, 2019

Any feedback on the revised version of this PR?

@bartlettroscoe
Copy link
Copy Markdown

bartlettroscoe commented Nov 7, 2019

CC: @tjfulle

I don't know the technical details of the implementation of this PR but I would be super excited if this type of feature could be merged and released in say v0.13.2. And there is even a automated test in this PR! (Thanks @tylerjereddy).

@tjfulle
Copy link
Copy Markdown
Contributor

tjfulle commented Nov 7, 2019

Just wanted to comment in support of some (any) method to get spack to look for the user configuration in a location other than ~/.spack. My PR #11951 has another alternative to accomplish the very same thing.

@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Nov 7, 2019

cc @gshipman

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Nov 27, 2019

@tylerjereddy @junghans @becker33: this is just a thought, but would it satisfy your use case better if we allowed some environments to be independent of everything except the Spack defaults?

So, for example, you could have:

spack::    # :: means do NOT include config from /etc, $spack/etc/spack, or ~/.spack
    ...

This way you could specify per-spack.yaml whether you want the environment to be independent of user/other settings on a machine.

I also think it's worthwhile to enable the user to customize whether a user's global config is factored in but I'm curious whether this would also fix your immediate problem.

@tylerjereddy
Copy link
Copy Markdown
Contributor Author

tylerjereddy commented Nov 27, 2019

@tgamblin I checked with Mike, and he didn't seem to think this would work--we want our spack instance to be able to specify that each user has a local insulated/unique ~/.spack_HASH that lives alongside whatever settings they may already have for their own ~/.spack (many users will not even realize they have a ~/.spack of their own).

Note that we have no plans to adopt environments in production anytime soon, so spack.yaml isn't really on our radar. I feel like the team discussed environments in detail & didn't find that they would suit our needs. Could be revisited later as they improve/evolve, though it would require substantial testing before adoption in production.

@tjfulle tjfulle mentioned this pull request Nov 27, 2019
Closed
@tjfulle
Copy link
Copy Markdown
Contributor

tjfulle commented Nov 27, 2019

#13916 is another PR to accomplish having the user config scope not be hard coded to ~/.spack. It was opened primarily to continue discussion of this topic.

@tgamblin tgamblin mentioned this pull request Jan 29, 2020
Open
@adamjstewart adamjstewart added the feature A feature is missing in Spack label Aug 23, 2020
@alalazo
Copy link
Copy Markdown
Member

alalazo commented May 10, 2022

Closing, since the underlying issue has been solved as documented here

@alalazo alalazo closed this May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@becker33 becker33 becker33 requested changes

@tgamblin tgamblin Awaiting requested review from tgamblin

@alalazo alalazo Awaiting requested review from alalazo

Assignees

No one assigned

Labels

feature A feature is missing in Spack

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Allow the user to customize the path for the ~/.spack folder ENH: control over ~/.spack handling

7 participants

@tylerjereddy @bartlettroscoe @tjfulle @tgamblin @alalazo @becker33 @adamjstewart