Skip to content

CI: AMD hip::device also for Fortran#2595

Merged
WeiqunZhang merged 2 commits intoAMReX-Codes:developmentfrom
ax3l:ci-recentHIPwFortran
Jan 21, 2022
Merged

CI: AMD hip::device also for Fortran#2595
WeiqunZhang merged 2 commits intoAMReX-Codes:developmentfrom
ax3l:ci-recentHIPwFortran

Conversation

@ax3l
Copy link
Copy Markdown
Member

@ax3l ax3l commented Jan 20, 2022 •
edited
Loading

Summary

By now (HIP 4.5), we should be able to link also against Fortran targets.

Let's remove work-arounds and ditch old HIP releases, so improve maintainability and avoid surprises.

Additional background

X-ref:

Checklist

The proposed changes:

  • fix a bug or incorrect behavior in AMReX
  • add new capabilities to AMReX
  • changes answers in the test suite to more than roundoff level
  • are likely to significantly affect the results of downstream AMReX users
  • include documentation in the code and/or rst files, if appropriate

@ax3l
CI: AMD hip::device also for Fortran
72ad01f 
By now (HIP 4.5), we should be able to link also against Fortran
targets.
@ax3l ax3l force-pushed the ci-recentHIPwFortran branch from 3c54722 to 72ad01f Compare January 20, 2022 20:24
@ax3l ax3l requested review from WeiqunZhang and sayerhs January 20, 2022 21:03
@ax3l ax3l changed the title CI: AMD hip::device also for Fortran [WIP] CI: AMD hip::device also for Fortran Jan 20, 2022
@ax3l
Copy link
Copy Markdown
Member Author

ax3l commented Jan 20, 2022 •
edited
Loading

Tested on OLCF Crusher with this environment: https://warpx.readthedocs.io/en/latest/install/hpc/crusher.html

[ 17%] Building CXX object _deps/localamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o
/sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.4.2-p3gb734ehpc3nfdqv3sxf7itxq5uvpmq/bin/ccache /opt/cray/pe/craype/2.7.13/bin/CC -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/ccs/home/huebl/src/amrex/Src/Base -I/ccs/home/huebl/src/amrex/Src/Base/Parser -I/ccs/home/huebl/src/amrex/Src/Boundary -I/ccs/home/huebl/src/amrex/Src/AmrCore -I/ccs/home/huebl/src/amrex/Src/LinearSolvers/MLMG -I/ccs/home/huebl/src/amrex/Src/Particle -I/ccs/home/huebl/src/warpx/build_crusher/_deps/localamrex-build -isystem /opt/rocm-4.5.2/llvm/lib/clang/13.0.0/include/.. -isystem /opt/rocm-4.5.2/hip/include -isystem /opt/rocm-4.5.2/hiprand/include -isystem /opt/rocm-4.5.2/rocrand/include -isystem /opt/rocm-4.5.2/rocprim/include -I/opt/rocm-4.5.2/include -O3 -DNDEBUG -m64 -munsafe-fp-atomics -x hip --offload-arch=gfx90a -std=c++17 -MD -MT _deps/localamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o -MF CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o.d -o CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o -c /ccs/home/huebl/src/amrex/Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp

fatal error: error in backend: Error while trying to spill SGPR4_SGPR5 from class SReg_64: Cannot scavenge register without an emergency spill slot!
clang-13: error: clang frontend command failed with exit code 70 (use -v to see invocation)
Cray clang version 13.0.0  (24b043d62639ddb4320c86db0b131600fdbc6ec6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/13.0.0/cce-clang/x86_64/share/../bin
clang-13: note: diagnostic msg: Error generating preprocessed source(s).
gmake[2]: *** [_deps/localamrex-build/Src/CMakeFiles/amrex.dir/build.make:1630: _deps/localamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o] Error 70
gmake[1]: *** [CMakeFiles/Makefile2:910: _deps/localamrex-build/Src/CMakeFiles/amrex.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

@ax3l
Copy link
Copy Markdown
Member Author

ax3l commented Jan 20, 2022 •
edited
Loading

If I add again -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false the compiler does not ICE, ICE, ICE.

Cray CC compilation on Crusher with ROCm 4.5.2:

  • no extra flags: ICE ❌
  • -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false: OK ✔️
  • -mllvm -amdgpu-early-inline-all=true: ICE ❌
  • -mllvm -amdgpu-function-calls=false: OK ✔️

Originally, I just extracted them from hip::device/hipcc as they did not properly escape those flags with SHELL: from deduplication ROCm/hip#2190

ax3l
ax3l commented Jan 20, 2022
View reviewed changes
Tools/CMake/AMReXParallelBackends.cmake
# note: already bumped to C++17 (cxx_std_17) or newer in AMReX_Config.cmake

# work-around for ROCm <=4.2
# https://github.com/ROCm-Developer-Tools/HIP/pull/2190
Copy link
Copy Markdown
Member Author

@ax3l ax3l Jan 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue tracked here:
ROCm/hipamd#12

@ax3l ax3l mentioned this pull request Jan 20, 2022
Open
@ax3l
Add Work-Around: AMD Flags for Cray
2fa6976 
Does not harm to add them unconditionally again.
@ax3l
Copy link
Copy Markdown
Member Author

ax3l commented Jan 20, 2022

Added the flags unconditionally now so that HPE/Cray CC also sees them. With that, Crusher continues to build.

@ax3l ax3l changed the title [WIP] CI: AMD hip::device also for Fortran CI: AMD hip::device also for Fortran Jan 20, 2022
@ax3l ax3l added the cleaning label Jan 21, 2022
@ax3l
Copy link
Copy Markdown
Member Author

ax3l commented Jan 21, 2022 •
edited
Loading

@WeiqunZhang I renamed the hip / HIP ROCm [email protected] C++17 [tests] build to hip / HIP ROCm Flang C++17 [tests], so you might need to change this in the settings of the repo for which test is required.

@WeiqunZhang WeiqunZhang merged commit be204df into AMReX-Codes:development Jan 21, 2022
@ax3l ax3l deleted the ci-recentHIPwFortran branch January 21, 2022 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WeiqunZhang WeiqunZhang WeiqunZhang approved these changes

@sayerhs sayerhs Awaiting requested review from sayerhs

Assignees

No one assigned

Labels

cleaning GPU install

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ax3l @WeiqunZhang