Multi-Dim Buildsystem Support by ax3l · Pull Request #3309 · AMReX-Codes/amrex

ax3l · 2023-05-12T01:47:43Z

Summary

This updates the CMake build system to allow to build multiple AMReX libraries at once, by building multiple, uniquely-named (lib)amrex_[1|2|3]d.[so|a|dll|lib] artifacts. Developers can request this by specifying a list of dims instead of a single one during configure time, e.g., -DAMReX_SPACEDIM="1;2;3".

Providing this functionality will significantly reduce deployment burdens for:

conda packages
spack packages
Python modules & packages
probably Julia packages (Johannes, blink twice when you read this)
HPC modules

because all dimension-related variants can be shipped at once and do not produce multiple, conflicting packages anymore.

Apps can use the same mechanism to pick up the dimensional variants and build multiple artifacts. (This is something we already do, but cannot generalize to package mangers just yet.)

Backward Compatibility

To stay backwards compatible, the last (or traditionally, only) passed dimensionality will define a few legacy artifacts:

the amrex::amrex/amrex CMake targets
the libamrex.[so|a]/amrex.[dll|lib] artifact (symlink)
the AMReX_Config.H file: defaults to the last dim in AMReX_SPACEDIM and defines AMREX_SPACEDIM via a public CLI define now

Additional background

I recently wrote an analysis piece lamenting that compile time software variants that define exclusive features are a productivity and usability challenge. One particular challenge we face in the shipping of WarpX and pyAMReX is that users (and developers) would like to seamlessly switch between the dims (1-3D) of our codes. Switching between compile-time and runtime description for such elementary workflows can and should be avoided.

Generally, this problem can be solved with more general interfaces in AMReX. Practically, this is a bit of work that we have not yet started.

Weiqun:

Other than EB and Linear Solvers, most of the amrex stuff built for 3d could work for 1d and 2d, including i/o.

Things this does not yet solve

As before, this does not solve namespace and/or symbol clashes. We should introduce unique name spaces and rework many globals in AMReX for this, but this is an orthogonal issue.

Checklist

The proposed changes:

fix a bug or incorrect behavior in AMReX
add new capabilities to AMReX
changes answers in the test suite to more than roundoff level
are likely to significantly affect the results of downstream AMReX users
include documentation in the code and/or rst files, if appropriate

JBlaschke · 2023-05-17T19:08:43Z

probably Julia packages (Johannes, blink twice when you read this)

@ax3l blinking...

This makes a lot of sense -- probably will also help Jackie with some of the things she wants to do.

WeiqunZhang · 2023-05-17T22:31:13Z

cmake_dependent_option( AMReX_EB "Build with Embedded Boundary support" OFF
   "NOT AMReX_SPACEDIM EQUAL 1" OFF )

does not work when AMReX_SPACEDIM is 1;2;3. It turns the EB option off. We may have to use option.

WeiqunZhang · 2023-05-17T22:41:03Z

--- a/Tools/CMake/AMReXOptions.cmake
+++ b/Tools/CMake/AMReXOptions.cmake
@@ -273,7 +273,7 @@ option( AMReX_AMRLEVEL  "Build AmrLevel class" ON )
 print_option( AMReX_AMRLEVEL )
 
 cmake_dependent_option( AMReX_EB "Build with Embedded Boundary support" OFF
-   "NOT AMReX_SPACEDIM EQUAL 1" OFF )
+   "2 IN_LIST AMReX_SPACEDIM OR 3 IN_LIST AMReX_SPACEDIM" OFF )
 print_option(AMReX_EB)
 
 cmake_dependent_option( AMReX_FORTRAN_INTERFACES "Build Fortran API" OFF

seems to work

ax3l · 2023-05-18T07:55:26Z

Thanks for spotting this!

I agree we can make this a regular option now, I added other fail-safes now for EB:
It simply warns and skips for the 1D geometry builds.

Warns and skips if selected for 1D

…dim-build

Likely untested interface changes that need to be updated in tests and covered in CI for 2D.

In this CI, we also run installation tests and the spack smoke test. This is a follow-up on AMReX-Codes#3309.

In this CI, we also run installation tests and the spack smoke test. This is a follow-up on #3309.

Fix two regressions from AMReX-Codes#3309 in the installed `AMReXConfig.cmake`: ``` CMake Error at /scratch/AMReX_RegTesting/amrex/builddir/lib/cmake/AMReX/AMReXConfig.cmake:257 (add_library): add_library cannot create ALIAS target "AMReX::amrex" because another target with the same name already exists. ``` and setting properties for newer CUDA.

ax3l · 2023-05-22T22:24:26Z

Tools/CMake/AMReXConfig.cmake.in

+list(LENGTH AMReX_SPACEDIM list_len)
+math(EXPR list_last "${list_len} - 1")
+list(GET AMReX_SPACEDIM ${list_last} AMReX_SPACEDIM_LAST)
+add_library(AMReX::amrex ALIAS AMReX::amrex_${AMReX_SPACEDIM_LAST}d)


This file seems to be allowed to be included multiple times (seen as a comment in the auto-generated AMReXTargets.cmake, which defines the actual ND targets). Thus, we need an include guard for add_library: #3324

ax3l · 2023-05-22T22:29:37Z

Tools/CMake/AMReXConfig.cmake.in

+if (@CMAKE_VERSION@ VERSION_GREATER_EQUAL 3.20 AND @AMReX_CUDA@)
   # CUDA architectures amrex was built for -- should we make
   set(AMREX_CUDA_ARCHS @AMREX_CUDA_ARCHS@ CACHE INTERNAL "CUDA archs AMReX is built for")
   set_target_properties(AMReX::amrex


Ooops, should use the non-alias targets. Fix in #3324

@drangara

## Summary Fix two regressions from #3309 in the installed `AMReXConfig.cmake`: ``` CMake Error at lib/cmake/AMReX/AMReXConfig.cmake:257 (add_library): add_library cannot create ALIAS target "AMReX::amrex" because another target with the same name already exists. ``` and setting properties for newer CUDA. ## Additional background Regression to #3309, spotted by @drangara, @ajnonaka, wdf et al. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate

It was broken in AMReX-Codes#3309, in which multi-D support was added.

It was broken in #3309, in which multi-D support was added.

ax3l added enhancement install labels May 12, 2023

ax3l force-pushed the topic-multi-dim-build branch 17 times, most recently from a1efbb9 to 6667b20 Compare May 15, 2023 00:57

ax3l changed the title ~~[WIP] Multi-Dim Buildsystem Support~~ Multi-Dim Buildsystem Support May 15, 2023

ax3l requested a review from WeiqunZhang May 15, 2023 00:57

ax3l force-pushed the topic-multi-dim-build branch 5 times, most recently from 3428f53 to 1567c82 Compare May 15, 2023 19:22

ax3l assigned WeiqunZhang May 15, 2023

ax3l force-pushed the topic-multi-dim-build branch from 1567c82 to 27b0bfb Compare May 16, 2023 00:12

This was referenced May 16, 2023

Multi-Dim Support AMReX-Codes/pyamrex#129

Merged

AMReX: 23.06+ Multi-Dim Support spack/spack#37695

Merged

EB: Always Option

15b937a

Warns and skips if selected for 1D

ax3l force-pushed the topic-multi-dim-build branch from dca7b20 to 15b937a Compare May 18, 2023 07:59

ax3l added 3 commits May 18, 2023 01:01

Merge remote-tracking branch 'mainline/development' into topic-multi-…

a39caee

…dim-build

Test: Parser2 Multi-Dim

7c9ce81

Start Fixing 2D Hypre Bugs

bafbf10

Likely untested interface changes that need to be updated in tests and covered in CI for 2D.

ax3l force-pushed the topic-multi-dim-build branch from 8780162 to bafbf10 Compare May 18, 2023 08:29

WeiqunZhang and others added 2 commits May 18, 2023 10:01

Finish fixing Hypre and PETSc for non-MultiFab

ab4ab42

PETSc: Same Type Update as for HYPRE

25a8d39

WeiqunZhang approved these changes May 18, 2023

View reviewed changes

WeiqunZhang merged commit ca70658 into AMReX-Codes:development May 18, 2023

ax3l deleted the topic-multi-dim-build branch May 18, 2023 23:57

ax3l mentioned this pull request May 20, 2023

Multi-Dim Buildsystem Support BLAST-WarpX/warpx#3933

Merged

2 tasks

WeiqunZhang added a commit to WeiqunZhang/amrex that referenced this pull request May 20, 2023

Add a new CI for multi-d builds

57812b5

In this CI, we also run installation tests and the spack smoke test. This is a follow-up on AMReX-Codes#3309.

WeiqunZhang mentioned this pull request May 20, 2023

Add a new CI for multi-d builds #3321

Merged

ax3l pushed a commit that referenced this pull request May 22, 2023

Add a new CI for multi-d builds (#3321)

f986296

In this CI, we also run installation tests and the spack smoke test. This is a follow-up on #3309.

ax3l mentioned this pull request May 22, 2023

Fix: AMReXConfig.cmake Multi-Dim #3324

Merged

5 tasks

ax3l commented May 22, 2023

View reviewed changes

ax3l mentioned this pull request May 24, 2023

Update: ABLASTR, AMReX, pyAMReX BLAST-ImpactX/impactx#371

Merged

1 task

baperry2 mentioned this pull request Jul 3, 2023

Redistribution move AMReX-Combustion/PeleC#647

Merged

WeiqunZhang added a commit to WeiqunZhang/amrex that referenced this pull request Jul 7, 2023

Fix Fortran module installation

72e37c1

It was broken in AMReX-Codes#3309, in which multi-D support was added.

WeiqunZhang mentioned this pull request Jul 7, 2023

Fix Fortran module installation #3411

Merged

baperry2 mentioned this pull request Jul 13, 2023

update amrex submod Exawind/amr-wind#876

Merged

asalmgren pushed a commit that referenced this pull request Jul 15, 2023

Fix Fortran module installation (#3411)

5a90c26

It was broken in #3309, in which multi-D support was added.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-Dim Buildsystem Support#3309

Multi-Dim Buildsystem Support#3309
WeiqunZhang merged 17 commits intoAMReX-Codes:developmentfrom
ax3l:topic-multi-dim-build

ax3l commented May 12, 2023 •

edited

Loading

Uh oh!

JBlaschke commented May 17, 2023

Uh oh!

WeiqunZhang commented May 17, 2023

Uh oh!

WeiqunZhang commented May 17, 2023

Uh oh!

ax3l commented May 18, 2023 •

edited

Loading

Uh oh!

ax3l May 22, 2023

Uh oh!

ax3l May 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ax3l commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Backward Compatibility

Additional background

Things this does not yet solve

Checklist

Uh oh!

JBlaschke commented May 17, 2023

Uh oh!

WeiqunZhang commented May 17, 2023

Uh oh!

WeiqunZhang commented May 17, 2023

Uh oh!

ax3l commented May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l May 22, 2023

Choose a reason for hiding this comment

Uh oh!

ax3l May 22, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ax3l commented May 12, 2023 •

edited

Loading

ax3l commented May 18, 2023 •

edited

Loading