config: CLI scopes should override environments#50853
Merged
Conversation
spackbot-app
bot
added
core
tgamblin
force-pushed
the
fix-cli-scope-order
branch
from
e161bf5 to
8ea01dd
Compare
Currently, active environments, from the CLI or from `$SPACK_ENV`, will override configuration set on the command line. This is confusing, as users would normally expect command line options to override environment options (e.g. SPACK_ENV). See [more discussion here](#49187 (comment)). This PR changes how we process configuration options on the CLI so that: 1. Any CLI option overrides an active environment from `SPACK_ENV` 2. CLI options that add config scopes (`-D`, `-C`, and `-e`) are now considered for scope precedence in the order they appear on the CLI. What this means is that if you write: * `spack -C config_dir`: `config_dir` will override any active enviornment. * `spack -C config_dir -e env`: `env` will override `config_dir` * `spack -e env -C config_dir -e env`: `config_dir` will override `env` And similar for `-D`. This makes the order much more obvious. This is a breaking change, in that historically environments have been higher precedence than config scopes. - [x] rework scope adding in `main.py` - [x] add an action to handle config from CLI - [x] add shell tests and unit tests Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
tgamblin
force-pushed
the
fix-cli-scope-order
branch
from
2cabe6b to
a1d1486
Compare
kwryankrattiger
approved these changes
Jun 10, 2025
Contributor
|
Expected results in spack-packages CI https://gitlab.spack.io/spack/spack-packages/-/pipelines/1120632 |
Member
This was merged without updates to the docs, so not sure it's obvious 😞 |
3 tasks
Member
|
I missed this PR back in June, but it basically recreates the issue that was solved by #48420 The gist of the original fix was to:
If we add an environment at CUSTOM priority, it's indistinguishable from other things added at CUSTOM priority and we need to do more bookkeeping to swap it in and out correctly. Given this was not documented, and that debugging #51459 was really hard, I'll submit a fix that uses ENVIRONMENT priority for environments. After all, that's what currently happens behind the scenes when the environment is loaded, with the exception that: says otherwise 😓 |
tgamblin
pushed a commit
that referenced
this pull request
Oct 28, 2025
…e used together (#51461) Fixes #51459. #50853 attempted to make configuration (envs and config scopes) specified on the CLI merge in CLI order, but did not quite succeed. Environments, when loaded, would still appear out of order in the scope stack. This fixes the issues and establishes, after much discussion, the ordering we want for configuration scopes specified on the CLI: 1. Ordering of scopes will be enforced *within* the same option on the CLI, e.g. bar is higher precedence in `-C foo -C bar`; 2. Ordering *across* options is counterintuitive, and the `-e` option (or the active environment via `SPACK_ENV`) should always be lower priority than `-C` or `-c` options; and 4. `-C` options have lower priority than `-c`. Environments are now added using `ConfigScopePriority.ENVIRONMENT` in all cases. Since they are the only entity using that scope priority, it's safe to activate and deactivate them. Documentation has been updated to reflect that the CUSTOM scopes have higher priority than ENVIRONMENT. --------- Signed-off-by: Massimiliano Culpo <[email protected]>
kshea21
pushed a commit
to kshea21/spack
that referenced
this pull request
Nov 4, 2025
…e used together (spack#51461) Fixes spack#51459. spack#50853 attempted to make configuration (envs and config scopes) specified on the CLI merge in CLI order, but did not quite succeed. Environments, when loaded, would still appear out of order in the scope stack. This fixes the issues and establishes, after much discussion, the ordering we want for configuration scopes specified on the CLI: 1. Ordering of scopes will be enforced *within* the same option on the CLI, e.g. bar is higher precedence in `-C foo -C bar`; 2. Ordering *across* options is counterintuitive, and the `-e` option (or the active environment via `SPACK_ENV`) should always be lower priority than `-C` or `-c` options; and 4. `-C` options have lower priority than `-c`. Environments are now added using `ConfigScopePriority.ENVIRONMENT` in all cases. Since they are the only entity using that scope priority, it's safe to activate and deactivate them. Documentation has been updated to reflect that the CUSTOM scopes have higher priority than ENVIRONMENT. --------- Signed-off-by: Massimiliano Culpo <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
Requires spack/spack-packages#119 to be merged first, or CI will break.
Currently, active environments, from the CLI or from
$SPACK_ENV, will override configuration set on the command line. This is confusing, as users would normally expect command line options to override environment options (e.g. SPACK_ENV).See more discussion here.
This PR changes how we process configuration options on the CLI so that:
SPACK_ENV-D,-C, and-e) are now considered for scope precedence in the order they appear on the CLI.What this means is that if you write:
spack -C config_dir:config_dirwill override any active enviornment.spack -C config_dir -e env:envwill overrideconfig_dirspack -e env -C config_dir -e env:config_dirwill overrideenvAnd similar for
-D. This makes the order much more obvious.This is a breaking change, in that historically environments have been higher precedence than config scopes.
main.py