amrex@23.06 +cuda cuda_arch=90 fails build: AMReX_BlockMutex.cpp.o: Error 139: Segmentation fault (core dumped)

### Steps to reproduce the issue

`amrex@23.06 +cuda cuda_arch=90` fails to build using:
* spack@develop (5c6c3b403bc4a8cd08b4e9303748933f3f79f5f7 from `Sun Jun 18 07:17:19 2023 -0400`)
* GCC 11.1.0
* CUDA 11.8.0
* Ubuntu 20.04, X86_64

Concrete spec: [amrex.spec.json.txt](https://github.com/spack/spack/files/11871178/amrex.spec.json.txt)

<details><summary>Concretization</summary>
<pre>
 -   amrex@23.06%gcc@11.1.0~amrdata+cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials~sycl~tiny_profile build_system=cmake build_type=Release cuda_arch=90 dimensions=3 generator=make precision=double arch=linux-ubuntu20.04-x86_64
[+]      ^cmake@3.26.3%gcc@11.1.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-ubuntu20.04-x86_64
[+]          ^ncurses@6.3%gcc@11.1.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]          ^openssl@1.1.1u%gcc@11.1.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-x86_64
[+]              ^ca-certificates-mozilla@2023-01-10%gcc@11.1.0 build_system=generic arch=linux-ubuntu20.04-x86_64
[+]              ^perl@5.36.0%gcc@11.1.0+cpanm+open+shared+threads build_system=generic arch=linux-ubuntu20.04-x86_64
[+]                  ^berkeley-db@18.1.40%gcc@11.1.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-x86_64
[+]                  ^bzip2@1.0.8%gcc@11.1.0~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-x86_64
[+]                  ^gdbm@1.23%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^zlib@1.2.13%gcc@11.1.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-x86_64
[+]      ^cuda@11.8.0%gcc@11.1.0~allow-unsupported-compilers~dev build_system=generic arch=linux-ubuntu20.04-x86_64
[+]          ^libxml2@2.10.3%gcc@11.1.0~python build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^libiconv@1.17%gcc@11.1.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]              ^xz@5.4.1%gcc@11.1.0+pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]      ^gmake@4.4.1%gcc@11.1.0~guile build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]      ^mpich@4.1.1%gcc@11.1.0~argobots~cuda+fortran~hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs~wrapperrpath build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu20.04-x86_64
[+]          ^findutils@4.9.0%gcc@11.1.0 build_system=autotools patches=440b954 arch=linux-ubuntu20.04-x86_64
[+]          ^libfabric@1.18.0%gcc@11.1.0~debug~kdreg build_system=autotools fabrics=rxm,sockets,tcp,udp arch=linux-ubuntu20.04-x86_64
[+]          ^libpciaccess@0.17%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^libtool@2.4.7%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^util-macros@1.19.3%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]          ^pkgconf@1.9.5%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]          ^yaksa@0.2%gcc@11.1.0~cuda~rocm build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^autoconf@2.69%gcc@11.1.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-ubuntu20.04-x86_64
[+]              ^automake@1.16.5%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^m4@1.4.19%gcc@11.1.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu20.04-x86_64
[+]                  ^diffutils@3.9%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                  ^libsigsegv@2.14%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]              ^python@3.7.15%gcc@11.1.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,f2fd060 arch=linux-ubuntu20.04-x86_64
[+]                  ^expat@2.5.0%gcc@11.1.0+libbsd build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                      ^libbsd@0.11.7%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                          ^libmd@1.0.4%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                  ^gettext@0.21.1%gcc@11.1.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                      ^tar@1.34%gcc@11.1.0 build_system=autotools zip=pigz arch=linux-ubuntu20.04-x86_64
[+]                          ^pigz@2.7%gcc@11.1.0 build_system=makefile arch=linux-ubuntu20.04-x86_64
[+]                          ^zstd@1.5.5%gcc@11.1.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]                  ^libffi@3.4.4%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                  ^libxcrypt@4.4.33%gcc@11.1.0~obsolete_api build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                  ^readline@8.2%gcc@11.1.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-x86_64
[+]                  ^sqlite@3.42.0%gcc@11.1.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]                  ^util-linux-uuid@2.38.1%gcc@11.1.0 build_system=autotools arch=linux-ubuntu20.04-x86_64
</pre></details>

Error:
```
$> spack install -f ./amrex.spec.json
...
==> Installing amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he
==> No binary for amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he found: installing from source
==> Using cached archive: /spack/var/spack/cache/_source-cache/archive/3b/3bddcb07cce3e65e06cac35005c30820d311ce47ae54b46e4af333fa272b236b.tar.gz
==> No patches needed for amrex
==> amrex: Executing phase: 'cmake'
==> amrex: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    '/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/gmake-4.4.1-lp2ol6q2quzz7sjtfkrxeczlsvzukkzk/bin/make' '-j16'

12 errors found in build log:
     134    [ 15%] Building CUDA object Src/CMakeFiles/amrex_3d.dir/Base/AMReX_Arena.cpp.o
     135    cd /tmp/root/spack-stage/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-build-5a63psa/Src && /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/cuda-11.8.0-e5qmi3yner
            ocnxupx3epemw25cezxkoo/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/usr/bin/g++ -DAMREX_SPACEDIM=3 --options-file CMakeFiles/amrex_3d.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++1
            7 --generate-code=arch=compute_90,code=[compute_90,sm_90] --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -
            Xcudafe --display_error_number --Wext-lambda-captures-this --Werror ext-lambda-captures-this --Werror cross-execution-space-call --use_fast_math --generate-line-info -Xcompiler=-Werror
            =return-type -Xcompiler -pthread -MD -MT Src/CMakeFiles/amrex_3d.dir/Base/AMReX_Arena.cpp.o -MF CMakeFiles/amrex_3d.dir/Base/AMReX_Arena.cpp.o.d -x cu -rdc=true -c /tmp/root/spack-stag
            e/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-src/Src/Base/AMReX_Arena.cpp -o CMakeFiles/amrex_3d.dir/Base/AMReX_Arena.cpp.o
     136    [ 16%] Building CUDA object Src/CMakeFiles/amrex_3d.dir/Base/AMReX_BArena.cpp.o
     137    cd /tmp/root/spack-stage/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-build-5a63psa/Src && /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/cuda-11.8.0-e5qmi3yner
            ocnxupx3epemw25cezxkoo/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/usr/bin/g++ -DAMREX_SPACEDIM=3 --options-file CMakeFiles/amrex_3d.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++1
            7 --generate-code=arch=compute_90,code=[compute_90,sm_90] --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -
            Xcudafe --display_error_number --Wext-lambda-captures-this --Werror ext-lambda-captures-this --Werror cross-execution-space-call --use_fast_math --generate-line-info -Xcompiler=-Werror
            =return-type -Xcompiler -pthread -MD -MT Src/CMakeFiles/amrex_3d.dir/Base/AMReX_BArena.cpp.o -MF CMakeFiles/amrex_3d.dir/Base/AMReX_BArena.cpp.o.d -x cu -rdc=true -c /tmp/root/spack-st
            age/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-src/Src/Base/AMReX_BArena.cpp -o CMakeFiles/amrex_3d.dir/Base/AMReX_BArena.cpp.o
     138    [ 17%] Building CUDA object Src/CMakeFiles/amrex_3d.dir/Base/AMReX_CArena.cpp.o
     139    cd /tmp/root/spack-stage/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-build-5a63psa/Src && /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.1.0/cuda-11.8.0-e5qmi3yner
            ocnxupx3epemw25cezxkoo/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/usr/bin/g++ -DAMREX_SPACEDIM=3 --options-file CMakeFiles/amrex_3d.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++1
            7 --generate-code=arch=compute_90,code=[compute_90,sm_90] --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -
            Xcudafe --display_error_number --Wext-lambda-captures-this --Werror ext-lambda-captures-this --Werror cross-execution-space-call --use_fast_math --generate-line-info -Xcompiler=-Werror
            =return-type -Xcompiler -pthread -MD -MT Src/CMakeFiles/amrex_3d.dir/Base/AMReX_CArena.cpp.o -MF CMakeFiles/amrex_3d.dir/Base/AMReX_CArena.cpp.o.d -x cu -rdc=true -c /tmp/root/spack-st
            age/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-src/Src/Base/AMReX_CArena.cpp -o CMakeFiles/amrex_3d.dir/Base/AMReX_CArena.cpp.o
  >> 140    Segmentation fault (core dumped)
  >> 141    make[2]: *** [Src/CMakeFiles/amrex_3d.dir/build.make:80: Src/CMakeFiles/amrex_3d.dir/Base/AMReX_BlockMutex.cpp.o] Error 139
     142    make[2]: *** Waiting for unfinished jobs....
  >> 143    Segmentation fault (core dumped)
  >> 144    make[2]: *** [Src/CMakeFiles/amrex_3d.dir/build.make:200: Src/CMakeFiles/amrex_3d.dir/Base/AMReX_Random.cpp.o] Error 139
  >> 145    Segmentation fault (core dumped)
  >> 146    make[2]: *** [Src/CMakeFiles/amrex_3d.dir/build.make:215: Src/CMakeFiles/amrex_3d.dir/Base/AMReX_DistributionMapping.cpp.o] Error 139
  >> 147    Segmentation fault (core dumped)
  >> 148    make[2]: *** [Src/CMakeFiles/amrex_3d.dir/build.make:245: Src/CMakeFiles/amrex_3d.dir/Base/AMReX_ForkJoin.cpp.o] Error 139
  >> 149    Segmentation fault (core dumped)
  >> 150    make[2]: *** [Src/CMakeFiles/amrex_3d.dir/build.make:275: Src/CMakeFiles/amrex_3d.dir/Base/AMReX_VisMF.cpp.o] Error 139
     151    make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-build-5a63psa'
  >> 152    make[1]: *** [CMakeFiles/Makefile2:217: Src/CMakeFiles/amrex_3d.dir/all] Error 2
     153    make[1]: Leaving directory '/tmp/root/spack-stage/spack-stage-amrex-23.06-5a63psa3tefjb6xr6idhjistjxwci7he/spack-build-5a63psa'
  >> 154    make: *** [Makefile:139: all] Error 2
```


### Error message

<details><summary>Error message</summary>
<pre>
see aboec
</pre></details>


### Information on your system

* **Spack:** 0.21.0.dev0 (5c6c3b403bc4a8cd08b4e9303748933f3f79f5f7)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-cascadelake
* **Concretizer:** clingo

### Additional information

[spack-build-env.txt](https://github.com/spack/spack/files/11871224/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/11871225/spack-build-out.txt)

@WeiqunZhang @asalmgren @etpalmer63 @ax3l @wspear @nchaimov

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[email protected] +cuda cuda_arch=90 fails build: AMReX_BlockMutex.cpp.o: Error 139: Segmentation fault (core dumped) #38556

Steps to reproduce the issue

Error message

Information on your system

Additional information

General information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[email protected] +cuda cuda_arch=90 fails build: AMReX_BlockMutex.cpp.o: Error 139: Segmentation fault (core dumped) #38556

Description

Steps to reproduce the issue

Error message

Information on your system

Additional information

General information

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions