AMReX: 23.06+ Multi-Dim Support by ax3l · Pull Request #37695 · spack/spack

ax3l · 2023-05-16T02:47:20Z

This updated the Spack package to allow to install AMReX, modules of AMReX in E4S deployments, and dependent packages, with support for multiple dimensions. Due to an upstream change in AMReX, we do not longer need to ship three, binary incompatible modules (resulting in three, incompatible HPC modules, resulting in three, incompatible variants per downstream app, etc. ...).

More background in this blog article: https://bssw.io/blog_posts/rethinking-software-variants

ax3l · 2023-05-23T03:01:54Z

I think I found a bug in variant(when=... - it always takes the default of the newer one, even if it is not applicable.
Mini-demo:

spack install amrex dimension=1,2,3 should:

either pick develop or
fail with Spack error

==> Error: multiple values are not allowed for variant "dimensions" in package "amrex"

But instead it tries to use 23.05, which is clearly not yet a multi variant, and fails during CMake configure (as it should).

ax3l · 2023-05-23T03:36:31Z

Looks like amrex@develop does the right thing:

$ spack spec amrex@develop
Concretized
--------------------------------
amrex@develop ... dimensions=1,2,3 ...

But not for older versions:

$ spack spec amrex

Concretized
--------------------------------
[email protected] ... dimensions=1,2,3

$ spack spec [email protected]

Concretized
--------------------------------
[email protected] ... dimensions=1,2,3

ax3l · 2023-06-09T17:59:08Z

Pushing the latest AMReX now, so this Spack bug will only will be weird for <=23.05, where people now need to pass dimensions=N with a single value.

For @spack/e4s et al., we should now ONLY build the default, which builds all dims at once (dimensions=1,2,3).

ax3l · 2023-06-12T18:14:25Z

@spack/e4s do you understand why the e4s-oneapi-build CI builder builds AMReX 23.06 (latest) and 22.12 instead of only the latest version 23.06?

I cannot find where this CI runner is configured to build 22.12, but in order to fix it please add dimensions=3 for <=23.05 releases of AMReX to enforce the old default, which this spack bug #38302 prevents to auto-pick.

eugeneswalker · 2023-06-13T04:35:49Z

@spack/e4s do you understand why the e4s-oneapi-build CI builder builds AMReX 23.06 (latest) and 22.12 instead of only the latest version 23.06?

I cannot find where this CI runner is configured to build 22.12, but in order to fix it please add dimensions=3 for <=23.05 releases of AMReX to enforce the old default, which this spack bug #38302 prevents to auto-pick.

Can you re-base this on develop? We updated the OneAPI CI stack on develop -- both the runner image and the Spack environment -- and if you re-base on develop, the older amrex will come somehow from the concretizer making a decision about what version to use for amrex +sycl. We can (1) re-base on develop and (2) simply add amrex +sycl dimensions=3 to the relevant spec line and it should be ok. What do you think?

ax3l · 2023-06-27T09:33:07Z

Rebased two weeks ago, shall I rebase again? :)

Do I need to do something for (2)?

eugeneswalker · 2023-06-27T18:02:29Z

It seems like the restriction of multi-dim to versions @23.06: is not working. The E4S OneAPI CI env is concretizing amrex +sycl to dimensions=1,2,3:

==> Concretized amrex+sycl
 -   o4vmisi  [email protected]%[email protected]~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials+sycl~tiny_profile build_system=cmake build_type=Release dimensions=1,2,3 generator=make precision=double arch=linux-ubuntu20.04-x86_64

Causing the obv error:

1 error found in build log:
     12    -- Check for working CXX compiler: /spack/lib/spack/env/oneapi/icpx - skipped
     13    -- Detecting CXX compile features
     14    -- Detecting CXX compile features - done
     15    -- CMake version: 3.26.3
     16    -- AMReX installation directory: /spack/opt/spack/linux-ubuntu20.04-x86_64/oneapi-2023.1.0/amrex-22.12-o4vmisib3mhy7tqrzmlthmnk7ektvzao
     17    -- Build type set by user to 'Release'.
  >> 18    CMake Error at Tools/CMake/AMReXOptions.cmake:28 (message):
     19      AMReX_SPACEDIM=1;2;3 is not allowed.  Must be one of 1;2;3
     20    Call Stack (most recent call first):
     21      CMakeLists.txt:84 (include)
     22
     23
     24    -- Configuring incomplete, errors occurred!

We can workaround this by appending a commit to this PR that explicitly specifies amrex +sycl dimensions=3 explicit in the E4S OneAPI CI Env:

Change this line:
https://github.com/spack/spack/blob/develop/share/spack/gitlab/cloud_pipelines/stacks/e4s-oneapi/spack.yaml#L229

To say amrex +sycl dimensions=3 and we will be good

This updated the Spack package to allow to install AMReX, modules of AMReX in E4S deployments and dependent packages with support for multiple dimensions. Due to an upstream change in AMReX, we do not longer need to ship three, binary incompatible package variants.

Work-around the auto-concretization to the multi-dim of `dimensions`, which only in 23.06+ became a multi-variant.

ax3l · 2023-07-27T07:18:37Z

Thank you @eugeneswalker, rebased and pushed accordingly.

eugeneswalker · 2023-07-27T16:12:57Z

The only failing CI job now is in E4S Cray RHEL CI stack w/ %cce@15

amrex/2uhnojf 23.07 cce@=15.0.1 linux-rhel8-zen4 E4S Cray

Unfortunately, the error trace is not very informative. I will try the build on this machine manually and post update here when I see what the problem is.

eugeneswalker · 2023-07-27T21:27:58Z

@spackbot run pipeline

spackbot-app · 2023-07-27T21:28:03Z

I've started that pipeline for you!

eugeneswalker · 2023-07-27T21:38:15Z

The only failing CI job now is in E4S Cray RHEL CI stack w/ %cce@15
* [amrex/2uhnojf 23.07 cce@=15.0.1 linux-rhel8-zen4 E4S Cray](https://gitlab.spack.io/spack/spack/-/jobs/7745604)
Unfortunately, the error trace is not very informative. I will try the build on this machine manually and post update here when I see what the problem is.

Ran this locally and the build completed just fine. Not sure what the issue is but I've restarted the pipeline. Let us see if we get a different result the second try...

…ailure can be resolved

eugeneswalker · 2023-07-29T11:02:28Z

Happy to merge whenever you think it is best @ax3l

* AMReX: 23.06+ Multi-Dim Support This updated the Spack package to allow to install AMReX, modules of AMReX in E4S deployments and dependent packages with support for multiple dimensions. Due to an upstream change in AMReX, we do not longer need to ship three, binary incompatible package variants. * [E4S] oneAPI AMReX < 23.06 Variant Work-around the auto-concretization to the multi-dim of `dimensions`, which only in 23.06+ became a multi-variant. * e4s cray rhel ci: temporarily disable amrex build until spurious ci failure can be resolved --------- Co-authored-by: eugeneswalker <[email protected]>

ax3l added ecp e4s labels May 16, 2023

spackbot-app bot requested review from WeiqunZhang, asalmgren and etpalmer63 May 16, 2023 02:47

spackbot-app bot added new-variant update-package labels May 16, 2023

ax3l force-pushed the amrex-multi-dim branch 4 times, most recently from 8c5ebda to 67ec6dc Compare May 23, 2023 02:49

alalazo self-assigned this May 29, 2023

spackbot-app bot added the new-version label Jun 9, 2023

ax3l mentioned this pull request Jun 9, 2023

Variant when=: Change of Default for Single/Multi-Variant Transition #38302

Closed

3 tasks

ax3l mentioned this pull request Jun 12, 2023

AMReX 23.06+: Multi-Dim Build E4S-Project/e4s#129

Closed

ax3l force-pushed the amrex-multi-dim branch from 2cdc902 to 812043c Compare June 22, 2023 20:44

ax3l force-pushed the amrex-multi-dim branch from 812043c to adad615 Compare June 27, 2023 09:34

ax3l added 2 commits July 27, 2023 00:15

[E4S] oneAPI AMReX < 23.06 Variant

ff37f34

Work-around the auto-concretization to the multi-dim of `dimensions`, which only in 23.06+ became a multi-variant.

ax3l force-pushed the amrex-multi-dim branch from adad615 to ff37f34 Compare July 27, 2023 07:18

spackbot-app bot added core PR affects Spack core functionality gitlab Issues related to gitlab integration labels Jul 27, 2023

e4s cray rhel ci: temporarily disable amrex build until spurious ci f…

d9d2b97

…ailure can be resolved

eugeneswalker approved these changes Jul 29, 2023

View reviewed changes

ax3l merged commit 7d44403 into spack:develop Aug 9, 2023

ax3l deleted the amrex-multi-dim branch August 9, 2023 18:25

balay mentioned this pull request Sep 30, 2023

AMReX build failure in xSDK CI xsdk-project/xsdk-issues#223

Closed

Conversation

ax3l commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l commented May 23, 2023

Uh oh!

ax3l commented May 23, 2023

Uh oh!

ax3l commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Jun 13, 2023

Uh oh!

ax3l commented Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Jun 27, 2023

Uh oh!

ax3l commented Jul 27, 2023

Uh oh!

eugeneswalker commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Jul 27, 2023

Uh oh!

spackbot-app bot commented Jul 27, 2023

Uh oh!

eugeneswalker commented Jul 27, 2023

Uh oh!

eugeneswalker commented Jul 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ax3l commented May 16, 2023 •

edited

Loading

ax3l commented Jun 9, 2023 •

edited

Loading

ax3l commented Jun 12, 2023 •

edited

Loading

ax3l commented Jun 27, 2023 •

edited

Loading

eugeneswalker commented Jul 27, 2023 •

edited

Loading