Skip to content

Acorn updates, March 2025#1571

Merged
AlexanderRichert-NOAA merged 23 commits intoJCSDA:developfrom
AlexanderRichert-NOAA:acorn_weeklybuild_mar25
Mar 20, 2025
Merged

Acorn updates, March 2025#1571
AlexanderRichert-NOAA merged 23 commits intoJCSDA:developfrom
AlexanderRichert-NOAA:acorn_weeklybuild_mar25

Conversation

@AlexanderRichert-NOAA
Copy link
Copy Markdown
Collaborator

@AlexanderRichert-NOAA AlexanderRichert-NOAA commented Mar 19, 2025 •
edited by climbfuji
Loading

Summary

General spack-stack updates:

  • 'esmf+python' as 'require' rather than 'prefer'.
  • By default, spack stack create env will now filter site/compilers.yaml to remove compilers with the same name but different versions (there is an optional flag for not doing this). There is also a new utility, filter_spack_compilers.py that allows users to filter compilers "manually" (inclusive, exclusive, and exclusive by compiler name/version).
  • In weekly builds, don't wipe out existing env dirs. I'm openminded as to whether this is actually a good idea.
  • Allow mapl duplicates in weekly builds.

Acorn-specific updates:

  • Enable MPI-based netcdf-{c,fortran} tests on Acorn.
  • Re-enable PBS Pro for weekly testing; build gh+deps on login nodes, everything else in PBS Pro.

Testing

Tested on Acorn.

Applications affected

Most (because of esmf:require:+python)

Systems affected

Acorn; maintainers that don't want to filter site/compilers.yaml can use the --keep-all-compilers flag

Dependencies

JCSDA/spack#527

Issue(s) addressed

Fixes #1570

Checklist

  • This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
  • These changes have been tested on the affected systems and applications.
  • All dependency PRs/issues have been resolved and this PR can be merged.

@AlexanderRichert-NOAA
require +python for esmf
149f3f8
@AlexanderRichert-NOAA
acorn: enable MPI tests for netcdf-{c,fortran}
bb2fccd
@AlexanderRichert-NOAA
allow recycling existing env dir
693b02b
@AlexanderRichert-NOAA
weeklybuild: ignore mapl dupl's
6d489b9
@AlexanderRichert-NOAA
acorn: weekly_build site updates
ba2aa29
@AlexanderRichert-NOAA
mv spack module ptr->testing
779caef
@AlexanderRichert-NOAA
fix gh/go logic for acorn weeklybuild
2ca86ed
@AlexanderRichert-NOAA
submodule pointer
1636aae
@AlexanderRichert-NOAA
submodule revert
6c86f42
@AlexanderRichert-NOAA
fix acorn weeklybuild install fn
bb56c94
@AlexanderRichert-NOAA
test submodule
0b78a93
@AlexanderRichert-NOAA
use $INSTALL_OPTS for gh install on cron runner node
6f4d4e1
@AlexanderRichert-NOAA
add filter_spack_compilers.py utility to customize the compiler list …
7d4aff9 
…in a given environment
@AlexanderRichert-NOAA
03_SetupEnv.sh: remove other version of same compiler name from site/…
d61e4dd 
…compilers.yaml
@AlexanderRichert-NOAA
filter_spack_compilers.py: use regular yaml module
23682eb
@AlexanderRichert-NOAA
by default, have 'create env' filter out other compiler versions of s…
a34c362 
…ame name
@AlexanderRichert-NOAA
revert 03_SetupEnv.sh compiler filtering
d2ed1b4
@AlexanderRichert-NOAA
fix compiler filter logic
8d814f9
@AlexanderRichert-NOAA
update acorn build.pbs
5b713c5
@AlexanderRichert-NOAA
Merge remote-tracking branch 'origin/develop' into acorn_weeklybuild_…
82daada 
…mar25
@AlexanderRichert-NOAA AlexanderRichert-NOAA marked this pull request as ready for review March 19, 2025 21:55
@AlexanderRichert-NOAA AlexanderRichert-NOAA requested review from RatkoVasic-NOAA, climbfuji, eap and rickgrubin-noaa and removed request for eap March 19, 2025 21:55
@rickgrubin-noaa
Copy link
Copy Markdown
Collaborator

rickgrubin-noaa commented Mar 19, 2025

Is the following a correct understanding?

  • when creating an env as is currently done: spack stack create env ... --compiler <name>, it will now be necessary for the --compiler arg to specify a <name> and <version>, e.g. oneapi@2024 if there is more than one entry for <name> in compilers.yaml ?
  • when creating an env as is currently done: spack stack create env ... --compiler <name> without specifying --keep-all-compilers, and multiple versions of <name>, e.g. oneapi@2024 and oneapi@2025, are both specified in compilers.yaml, then <name>@<version> not specified as an arg to --compiler is filtered ?
  • if --keep-all-compilers is specified when creating an env, one can filter out a compiler after the fact with the util/filter_spack_compilers.py utility ?

@AlexanderRichert-NOAA
Copy link
Copy Markdown
Collaborator Author

AlexanderRichert-NOAA commented Mar 19, 2025 

* when creating an env as is currently done: `spack stack create env ... --compiler <name>`, it will now be necessary for the `--compiler` arg to specify a `<name>` **_and_** `<version>`, e.g. `oneapi@2024` if there is more than one entry for `<name>` in `compilers.yaml` ?

No- If no version is specified by the --compiler flag, site/compilers.yaml will not be modified.

* when creating an env as is currently done: `spack stack create env ... --compiler <name>` without specifying `--keep-all-compilers`, and multiple versions of `<name>`, e.g. `oneapi@2024` and `oneapi@2025`, are both specified in `compilers.yaml`, then `<name>@<version>` **_not_** specified as an arg to `--compiler` is filtered ?

Correct (assuming that some version was provided with the --compiler flag)

* if `--keep-all-compilers` is specified when creating an env, one can filter out a compiler after the fact with the `util/filter_spack_compilers.py` utility ?

Correct

climbfuji
climbfuji requested changes Mar 20, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@climbfuji climbfuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions/requests

  1. Please update the documentation. We have documentation for utilities, for the spack-stack extension, ... - all in readthedocs. Given that this is a little more complicated than some of the other options, we should document it properly.
  2. Do we need the extra utility filter_spack_compilers.py? I'd say it's enough to do this when setting up the environment, and if you forget then start over?

@AlexanderRichert-NOAA
Copy link
Copy Markdown
Collaborator Author

AlexanderRichert-NOAA commented Mar 20, 2025

  1. I added a couple bits of documentation, let me know if there's somewhere else that it should get mentioned or if you think it needs further elaboration.
  2. Probably not? I intend to maintain a copy of it elsewhere for use outside of spack-stack, so I won't add it unless others would find it useful.

climbfuji
climbfuji approved these changes Mar 20, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@climbfuji climbfuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge this and resolve the conflicts later when merging release/1.9.0 into develop.

@AlexanderRichert-NOAA AlexanderRichert-NOAA merged commit 8faa949 into JCSDA:develop Mar 20, 2025
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@climbfuji climbfuji climbfuji approved these changes

@rickgrubin-noaa rickgrubin-noaa rickgrubin-noaa approved these changes

@eap eap Awaiting requested review from eap

@RatkoVasic-NOAA RatkoVasic-NOAA Awaiting requested review from RatkoVasic-NOAA

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

'spack stack create env' automatically filter unused compilers from site/compilers.yaml?

3 participants

@AlexanderRichert-NOAA @rickgrubin-noaa @climbfuji