segfault when importing numpy after reinitializing Python #11925
Description
I embedded Python into a C program. At certain points in time, the C program resets the interpreter by calling Py_Finalize + Py_Initialize. When the evaluated code imports NumPy before and after the reset, then the second import fails with a segmentation fault.
Reproducing code example:
At first the example C program initializes Python, imports NumPy, and then finalizes Python.
Then it reinitializes Python and imports NumPy again. When importing NumPy
for the second time, the import fails with a segmentation fault.
Tested with Debian 9, Python 3.7.0 valgrind friendly debug build, NumPy debug build of latest
version from github (1.16.0.dev0+e796449). The bug reproduces on Mac (tested with Python 3.5.1,
NumPy 1.11.0) and Windows (Tested with Python 3.7.0 from Python.org, NumPy 1.14.5).
/******************************************************************************
Test: Initialize NumPy before and after Python reinitialization
******************************************************************************/
#include "Python.h"
int main(int arc, char *argv[]) {
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
exit(1);
}
Py_SetProgramName(program);
// Initialize Python, import NumPy, finalize Python. First time.
Py_Initialize();
PyRun_SimpleString("import sys; print('Python', sys.version)");
PyRun_SimpleString("import numpy; print('Numpy', numpy.__version__)");
Py_Finalize();
// Initialize Python, import NumPy, finalize Python. Second time.
Py_Initialize();
PyRun_SimpleString("import numpy;"); // <-- SEGMENTATION FAULT.
Py_Finalize();
PyMem_RawFree(program);
return 0;
}Error message:
Segmentation fault on evaluation of "import numpy" after reinitialization of Python. Output of valgrind:
==4346== Memcheck, a memory error detector
==4346== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4346== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==4346== Command: ./main
==4346==
Python 3.7.0 (default, Aug 27 2018, 19:39:47)
[GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449
==4346== Invalid read of size 8
==4346== at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346== by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346== by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346== by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346== by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346== by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346== by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346== by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346== by 0x4FC42FD: call_function (ceval.c:4586)
==4346== by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346== by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346== by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346== by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346== Address 0x1a is not stack'd, malloc'd or (recently) free'd
==4346==
==4346==
==4346== Process terminating with default action of signal 11 (SIGSEGV)
==4346== Access not within mapped region at address 0x1A
==4346== at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346== by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346== by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346== by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346== by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346== by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346== by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346== by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346== by 0x4FC42FD: call_function (ceval.c:4586)
==4346== by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346== by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346== by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346== by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346== If you believe this happened as a result of a stack
==4346== overflow in your program's main thread (unlikely but
==4346== possible), you can try to increase the size of the
==4346== main thread stack using the --main-stacksize= flag.
==4346== The main thread stack size used in this run was 8388608.
==4346==
==4346== HEAP SUMMARY:
==4346== in use at exit: 6,190,252 bytes in 34,347 blocks
==4346== total heap usage: 215,244 allocs, 180,897 frees, 45,494,245 bytes allocated
==4346==
==4346== LEAK SUMMARY:
==4346== definitely lost: 144 bytes in 2 blocks
==4346== indirectly lost: 0 bytes in 0 blocks
==4346== possibly lost: 6,055,624 bytes in 31,510 blocks
==4346== still reachable: 134,484 bytes in 2,835 blocks
==4346== suppressed: 0 bytes in 0 blocks
==4346== Rerun with --leak-check=full to see details of leaked memory
==4346==
==4346== For counts of detected and suppressed errors, rerun with: -v
==4346== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed
Numpy/Python version information:
Python 3.7.0 (default, Aug 27 2018, 19:39:47) [GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449