It's weird how the CI keeps trying to build for gfx701. I suspect it's because the amdgpu_target variant is underspecified in some specs. I've opened #34591 to try to prevent some of those weird failures.

@spackbot rerun pipeline

edit: opps. wrong PR.

I've started that pipeline for you!

The ROCmPackage default of (amdgpu_target=none) is different from how the ROCm libraries have worked historically, which is breaking libraries that depend on the hip mathlibs. I'm going to mark this as a draft until I know how to deal with that.

the new amdgpu_target variant is required (as that's how ROCmPackage works).

This is unfortunate. It means that spack spec hipsolver is resolved as hipsolver +cuda since amdgpu_target=none is the default, is illegal, and is sticky.

@haampie, @renjithravindrankannath, @srekolam, please review these changes.

I've made some changes so that this PR doesn't have to wait on #35233 and #35316. The custom disjoint set that is needed for this is a bit ugly, but the awfulness is at least contained to these libraries. I explicitly avoided extracting it into a function to ensure that it doesn't get used elsewhere. We can remove it once the amdgpu_target is properly detected.

The custom disjoint set is exactly the same as the amdgpu_target variant from ROCmPackage, but it adds the value of auto and sets it as the default. This works with the rocm mathlibs, which also have a default value of auto. This is how my proposed change preserves the existing default behaviour for specs that don't specify the amdgpu_target.

Once there is proper amdgpu_target detection logic in spack (both for concretizing specs and for external find), then the auto variant value can be dropped from rccl, rocalution, roccblas, rocfft, rocprim, rocrand, rocsolver, rocsparse, rocthrust, rocmma, hipblas, hipsolver, hipsparse, hipfft and hipcub.

*poke*

@cgmb , Please excuse my delayed response. I tried [email protected] +cuda , [email protected] +cuda, [email protected] +cuda, [email protected] +cuda and [email protected] +cuda. hipblas, hipfft, hipsolver were built successfully. But hipcub & hipsparse were failing.
hipcub-5.4.3-cuda-spack-build-out.txt
hipsparse-5.4.3-cuda-spack-build-out.txt

hip-targets-release.cmake and hip-targets.cmake are missing under the lib/cmake/hip/ path

[xyz@ixt-rack-104 spack]$ ls -la opt/spack/linux-ubuntu20.04-skylake_avx512/gcc-9.4.0/hip-5.4.3-ltezppdane3i6xpri3gdl2f7rvp4ucak/lib/cmake/hip/
total 68
drwxr-sr-x. 3 root root  4096 Mar 28 01:31 .
drwxr-sr-x. 3 root root  4096 Mar 28 01:31 ..
drwxr-sr-x. 2 root root  4096 Mar 28 01:31 FindHIP
-rw-r--r--. 1 root root 33450 Jan 20 09:05 FindHIP.cmake
-rw-r--r--. 1 root root 13266 Mar 28 01:29 hip-config.cmake
-rw-r--r--. 1 root root  2890 Mar 28 01:29 hip-config-version.cmake

@cgmb sorry for being late on this. I looked at your changes and i took your changes on my local spack repository and build hipfft+ cuda and [email protected]. It built both versions successfully. A couple of observations with the hip. When i execute the hipcc and hipconfig from both the installed prefix paths of hip + rocm and hip+cuda, i see it always points to cuda driver .. PR-34586.txt same with hipconfig too.. Just trying to check with you on this.

You need to spack load hip +rocm or spack load hip +cuda to ensure your environment is correctly configured before running hipconfig or hipcc.

hipcub-5.4.3-cuda-spack-build-out.txt

I can't reproduce this, but it looks like a bug in hipcub. The hip-config.cmake file is getting picked up by this call to find_package(hip QUIET CONFIG ...). I've added a couple patches in d2fcda2 to try to fix it for [email protected]: +cuda. However, it works for me even without the patches. So, they're not strictly necessary in some environments.

hipsparse-5.4.3-cuda-spack-build-out.txt

This is a bug in hipsparse. It doesn't support newer versions of CUDA. I've backported the fix in 9fc769d.

hip-targets-release.cmake and hip-targets.cmake are missing under the lib/cmake/hip/ path

That is normal. HIP does not yet support config mode search on the NVIDIA platform.

@cgmb sorry for being late on this. I looked at your changes and i took your changes on my local spack repository and build hipfft+ cuda and [email protected]. It built both versions successfully. A couple of observations with the hip. When i execute the hipcc and hipconfig from both the installed prefix paths of hip + rocm and hip+cuda, i see it always points to cuda driver .. PR-34586.txt same with hipconfig too.. Just trying to check with you on this.

You need to spack load hip +rocm or spack load hip +cuda to ensure your environment is correctly configured before running hipconfig or hipcc.

hipcub-5.4.3-cuda-spack-build-out.txt

I can't reproduce this, but it looks like a bug in hipcub. The hip-config.cmake file is getting picked up by this call to find_package(hip QUIET CONFIG ...). I've added a couple patches to try to fix it in [email protected]: +cuda. However, it works for me even without the patches. So, they're not strictly necessary in some environments.

hipsparse-5.4.3-cuda-spack-build-out.txt

This is a bug in hipsparse. It doesn't support newer versions of CUDA. I've backported the fix.

hip-targets-release.cmake and hip-targets.cmake are missing under the lib/cmake/hip/ path

That is normal. HIP does not yet support config mode search on the NVIDIA platform.

Thanks @cgmb . hipcub and hipsparse built successfully for cuda variant.