Add RCCL Static Lib Creation with -fgpu-rdc#260
Conversation
aaronenyeshi
requested review from
saadrahim,
stanleytsang-amd,
wenkaidu and
yxsamliu
|
Tested that this patch creates |
|
Do static builds work in the standard mainline docker containers or do we need a separate static container? |
|
I want to update our CI's static tests so that this PR is correctly tested in the math library CI system. |
This patch was tested in the standard mainline docker container 3565. I'm not sure about the separate static containers. |
| # through -l and -L instead of command line input. | ||
| if(BUILD_STATIC) | ||
| add_dependencies(UnitTests rccl) | ||
| target_link_libraries(UnitTests PRIVATE dl rt numa -lrccl -L${CMAKE_BINARY_DIR}) |
There was a problem hiding this comment.
Does rccl itself depend on these libraries? If so, shouldn't they be specified through target_link_libraries(rccl INTERFACE ...) ?
There was a problem hiding this comment.
Right, RCCL depends on these libraries, but normally it should have been brought in when we set link to hip::device (which includes them from either hsa-runtime::hsa-runtime64 or other deps). But I couldn't figure out how to force CMake to resolve something like hip::device into its list of link deps.
Also, we may need to support librccl.a as an input file instead of using -L and -l?
There was a problem hiding this comment.
It looks like I need to explicitly put the static libraries into the PRIVATE libraries for UnitTests. I'm not sure how to work around it, but it doesn't seem like its picking up all the dependencies in static mode. I checked hip::device, and the dependency on hsa-runtime64::hsa-runtime64 is a PRIVATE one.
RCCL uses -fgpu-rdc to compile its source objects. When linking the RCCL static library, the link and archive step must do through hipcc and uses the flag --emit-static-lib. When compiling UnitTests, the librccl.a must be consumed through -l and -L.
aaronenyeshi
force-pushed
the
adding-static-lib
branch
from
60ae1e1 to
463460f
Compare
@saadrahim - this new patch I pushed was tested in the static container. And I ran some of the 2GPU unit tests on my machine. I've changed some of the syntax to support cmake 3.10 which is used in the static images. |
## Summary Add `-fgpu-rdc` flags to HIP if requested via `AMReX_GPU_RDC`. Modernize HIP logic with recommended targets that avoid the flaky `hipcc` compiler scripts: https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake Add support for AMDs `clang++`/`clang` compiler for HIP instead of using the legacy `hipcc` perl wrapper as C++ compiler. This also increases support towards Cray compiler wrappers, which also refer to `clang++`/`clang` underneath (Spock/OLCF). Close #1688 ## Additional background Relocatable-device-code (RDC) flags are needed for `extern` device variable support (for codes that use global variables on device). Also needed when linking with Ascent. Follow-up to #2029 With HIP GPU RDC, static libs emitting & linking does get more fancy: ROCm/rccl#260
4 participants
RCCL uses -fgpu-rdc to compile its source objects. When linking
the RCCL static library, the link and archive step must do through
hipcc and uses the flag --emit-static-lib. When compiling
UnitTests, the librccl.a must be consumed through -l and -L.