When I attempt to import numpy on NERSC Cori KNL Nodes (68 physical cores, 4*68 = 272 hyperthreads) an abort that crashes the program occurs. This apparently coming from OpenBLAS, however I think this is a numpy issue since it occurs in import numpy, I've compiled Numpy with default options, and numpy is the one using /configuring OpenBLAS in this case.

I can work around this by export OMP_NUM_THREADS=1 however this work around is problematic because there is other software that is parallelized using OpenMP that will be called from my Python script and this other code doesn't have the 256 thread limit that OpenBLAS does.

A solution might be for numpy to configure OpenBLAS to use a fixed number of threads by default. In my experience using more than the number of physical cores rarely provides a speed up in numerical codes. Further, in MPI parallel codes one must take into account the number of physical cores per MPI rank. Eg if one ran 68 MPI ranks per node and in each ramk OpenBLAS tried to use 272 cores, overall performance would be degraded. For those reasons, configuring OpenBLAS to use 1 thread would likely be the best default value.

An alternate solution could be a numpy specific hook to set the number of OpenBLAS threads as used by numpy. This way it will not interfere with other codes that use OpenMP for parallelism.

Yet another solution, would be for OpenBLAS not to call abort, however if it tried to use 256 cores on these systems less than optimal performance would be achieved.

Reproducing code example:

$python3
Python 3.8.5 (default, Nov 18 2020, 06:04:22) 
[GCC 8.3.0 20190222 (Cray Inc.)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy

OpenBLAS Warning : The number of CPU/Cores(272) is beyond the limit(256). Terminated.

Error message:

OpenBLAS Warning : The number of CPU/Cores(272) is beyond the limit(256). Terminated.

NumPy/Python version information:

Python 3.8.5 (default, Nov 18 2020, 06:04:22) 
[GCC 8.3.0 20190222 (Cray Inc.)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, numpy; print(numpy.__version__, sys.version)
1.19.1 3.8.5 (default, Nov 18 2020, 06:04:22) 
[GCC 8.3.0 20190222 (Cray Inc.)]