Allow user to customize the config path#23760
Conversation
lib/spack/spack/paths.py
Outdated
|
|
||
| #: User configuration location | ||
| user_config_path = os.path.expanduser('~/.spack') | ||
| __SPACK_USER_CONFIG = os.environ.get('SPACK_USER_CONFIG') |
There was a problem hiding this comment.
I'm not sure that this should be environment dependent, why is it insufficient to set it via etc/config.yaml?
There was a problem hiding this comment.
Is it currently possible to set it in etc/config.yaml? My understanding was that ~/.spack is used unconditionally at present. Having it be a setting you can get from either etc, or from an environment, would be great and I wouldn't mind making that change, but wanted to get this up here to spark this kind of discussion. Thoughts @alalazo, @tgamblin?
There was a problem hiding this comment.
Yes you are correct ~/.spack is currently unconditionally used. This bit us multiple times (... we (mostly) amputated the user config scope) and I love to have it configurable.
But I'd prefer to have it settable via the config file rather than the environment (I have to adapt the config anyways and the environment is uncontrolled, from the perspective of the spack repo).
There was a problem hiding this comment.
I largely agree, but we have a control inversion, or at least ordering, problem. In principle the user config overrides the environment, repo and system configs. I'm thinking we have a few options:
- add a way to configure how spack should order configs, something like XDG_CONFIG_DIRS, so you can order how all configurations should be applied, example:
$env:$repo:$user:/some/local/config/path:$systemto give env highest priority over repo and user- Apply this with an env var
- Apply this with a new file in home or XDG_CONFIG_HOME or similar, that can be used to set up environment before any normal config is used, think .zshrc loading before the zsh config path is searched for .zshenv so .zshenv can live anywhere
- make this setting, or a file containing it, "special" so that one can make a truly local repo without need for environment or user config to say so
There was a problem hiding this comment.
Having the value of the user config path settable in etc/config.yaml will likely not work with a shared instance of Spack since not all users will have write privileges. The environment variable seems the easiest way to implement this behavior.
There was a problem hiding this comment.
These lines could be simplified as:
user_config_path = os.getenv("SPACK_USER_CONFI_PATH", os.path.expanduser("~/.spack"))There was a problem hiding this comment.
Good point, that's actually not quite right but moving the expanduser outside the getenv does what I want, thanks @tjfulle!
As to etc/config.yaml, I agree the env var is most flexible for a user of a shared spack, but it means that the owner of that specific installation of spack can't set a default for the usage of it. I can see uses either direction, but the env var is simpler so going for that first makes a lot of sense to me.
|
This is essentially a duplicate of #11951 which also links many related issues. #13196 is another more complicated solution. Bottom line, there needs to be a way to change the location of the user config path. As @trws points out in #23760, the solution of the user defining a single environment variable to the config path is far less complicated than any configuration based approach and is a de-facto standard way of customizing software on HPC systems. |
|
Yup, it’s intentionally a re-hash of that territory. At this point I
don’t care so much what the solution is as long as we have one. It
would be nice to be able to make a “local” spack repo that didn’t
require any other config to be treated as local, some way to add
something to the directory to make that happen, but there’s no obvious
clear way to do that in the context of the existing system. Could
always make it adding a file to the prefix I guess, add a file like
`$prefix/STANDALONE` to make it happen maybe, but the environment
variable is at least a functional partial solution.
…On 21 May 2021, at 17:45, Tim Fuller wrote:
This is essentially a duplicate of #11951 which also links many
related issues. #13196 is another more complicated solution. Bottom
line, there needs to be a way to change the location of the user
config path. As @trws points out in
https://urldefense.us/v3/__https://github.com/spack/spack/pull/23760?_pjax=*23js-repo-pjax-container*discussion_r636246001__;JSM!!G2kpM7uM-TzIFchu!k3dfxOsJvyqc2ODsijkJAgOGZhNMtnTbH99luS2YTf37JrBJnsq7qg_rK0Wy8K2F5Q$
, the solution of the user defining a single environment variable to
the config path is far less complicated than any configuration based
approach and is a de-facto standard way of customizing software on HPC
systems.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://urldefense.us/v3/__https://github.com/spack/spack/pull/23760*issuecomment-846322474__;Iw!!G2kpM7uM-TzIFchu!k3dfxOsJvyqc2ODsijkJAgOGZhNMtnTbH99luS2YTf37JrBJnsq7qg_rK0XxpQNfDw$
|
|
@trws - I prefer the environment variable approach, it's simple and standard in the Linux workd. I like to download a single instance of Spack and use it across projects, loading only a module that sets the correct Spack configuration path variable. It would be nice if this were standard so I didn't have to apply patches to enable the behavior. |
For some of my main cases, I want config to be rooted in whatever spack I'm currently using if possible, this allows that to be set. Need to think through what the fallback is if that's not possible because it's a shared spack not owned by the user, but this is on the way.
|
I'd really like to see an environment variable for spack.paths.system_etc_path. I'd like to be able to define system compilers, mpi, etc. for all users located on a shared file system not at a hard coded /etc/ which is not great for HPC cluster deployments. It should be as simple as this in paths.py: |
|
Force re-running the required checks, especially the apparently hung pipeline that's trying to run using a UO runner. |
|
Bump. @tldahlgren @tgamblin is this being looked at? It's extremely important to many of us who use Spack in our app distribution models that the installation is not sucking up configurations from the user's home directory. |
There appears to be outstanding feedback so it would be good to double-check with commenters. There is also a failing required check. We could tell CI to re-run the pipeline |
| os.environ.get( | ||
| 'SPACK_USER_CONFIG', | ||
| '~/.spack'))) |
There was a problem hiding this comment.
Isn't it better to make an empty value default to ~/.spack too?
os.environ.get('SPACK_USER_CONFIG') or '~/.spack'.
Also I would prefer SPACK_USER_CONFIG_PATH. It's super long, but it can only mean one thing.
There was a problem hiding this comment.
I prefer SPACK_USER_CONFIG_PATH as well.
As to making an empty value default to ~/.spack - I suppose if the user took the effort to export SPACK_USER_CONFIG_PATH= they did so for a reason?
There was a problem hiding this comment.
To some people that means the variable was "unset". I'm not very much attached to that opinion though
7 participants
This is the most basic possible way to do this. XDG support might be nice, etc. etc. but at a minimum this seems to allow a user to specify where they want the user_config_directory to reside, and makes it possible to support having separate ones a bit more easily on shared filesystems or with multiple instances of spack.