Crash when importing numpy from the Python C-API after calling Py_Finilize()

Hi,

We are getting a crash in the numpy code when importing numpy after Py_Finalize() is called. If we comment out the call to Py_Finalize() the second import will be successful.

#include "python2.7/Python.h"

int RunPythonScript(const char * pszScript)
{
    Py_Initialize();

    int nResult = PyRun_SimpleStringFlags(pszScript, NULL);

    Py_Finalize();

    return nResult;
}

int main(void)
{
    RunPythonScript("import numpy");

    printf("First import of numpy ran successfully!\n");

    RunPythonScript("import numpy");

    printf("Second import of numpy ran successfully!\n");

    return 0;
}

I compiled this example using GCC: gcc -o numpy_crash numpy_crash.cpp -lpython2.7

This simple example should not crash. My guess is that Python will destroy some global objects when Py_Finalize() is called. But these global objects are still referenced in the numpy code. You would need to register an atexit function using Py_AtExit() to reinitialize any global objects to NULL so that they are re-created on subsequent import of numpy.

This bug is most likely the same bug as #656 and #3961. But we are only running a single Python interpreter. We call Py_Finalize() every time a user runs a new Python script so that the variables they defined on the previous runs are no longer available in the Python environment.

Regards,

Phelippe Neveu

I have encountered the same issue when attempting to embed Python in an application as well running Python 3.5.

I don't think anyone would object to receiving a fix for this, but it would require a tiresome whole-codebase audit to find all the places numpy caches state, which is unlikely to be a priority for most contributors. So you should probably assume that it's not going to happen unless you do it.

@njsmith

@njsmith just FYI, i spent ~15 minutes looking at this and this issue has been reported many times over past 10 years in various threads with no clear resolution:

pythonnet/pythonnet#595

Perhaps numpy could provide its own finalization/cleanup method so that this can be called before finalizing the Python interpreter?

It seems to me nobody has *really* needed a fix for this, since nobody has come forward with work towards it or a patch.

Our current work around is to not call Py_Finalize() at all. I also added some code that frees attributes from the main module so that the variables defined in a previous run are not available in a future run.

`

PyObject * poMainModule = PyImport_AddModule("__main__");

PyObject * poAttrList = PyObject_Dir(poMainModule);

PyObject * poAttrIter = PyObject_GetIter(poAttrList);

PyObject * poAttrName;

while ((poAttrName = PyIter_Next(poAttrIter)) != NULL)
{
    std::string oAttrName = GetUnicodeString(poAttrName);

    // Make sure we don't delete any private objects.
    if (!StartsWith(oAttrName, "__") || !EndsWith(oAttrName, "__"))
    {
        PyObject * poAttr = PyObject_GetAttr(poMainModule, poAttrName);

        // Make sure we don't delete any module objects.
        if (poAttr && poAttr->ob_type != poMainModule->ob_type)
            PyObject_SetAttr(poMainModule, poAttrName, NULL);

        Py_DecRef(poAttr);
    }

    Py_DecRef(poAttrName);
}

Py_DecRef(poAttrIter);

Py_DecRef(poAttrList);

`

Regards,

Phelippe

Hi,
I experience exactly the same behavior as Phelippe. I embed execution of python scripts into a C application. If numpy is imported in the script, the application crashes the second time the script is executed.
Each time a script is run, pyInitialize and pyFinalize are called.
The only way to make it work is to stop and restart our application before running the script a second time which is not the way it should work.
It would be nice to have a fix.
Regards,
Alain

I have same problem as Phelippe.
I need to unload/reload python scripts (python version 2.7.14) using numpy library (it's a mathematical simulation application).
If Phelippe can explain better his workaround I can try to use it in the meantime:

PyObject_Dir(poMainModule) => need to get poMainModule before
GetUnicodeString,StartsWith,EndsWith => are customized functions or defined in some Windows or python library?

Phelippe's workaround basically lists all the objects accessible in the main module and deletes them one by one provided that they:

do not start or end with '__'
are not of module type

One way of getting the main module would be:

 PyObject* poMainModule = PyImport_ImportModule("__main__");
 if (poMainModule != NULL) {
         // Rest of code here...
 }

The GetUnicodeString, StartsWith and EndsWith are probably custom functions as I couldn't find any Windows or Python documentation on them (except in the .NET framework doc which should not be relevant here).

EDIT:

You might want to take a look at the PyString_AsString function from the Python C API

One of the reason why we wanted to call Py_Finalize() was to make sure that if the user defined variables in a previous run, that they were not available in subsequent runs. The only way I found to do this is to loop through all of the main module's attributes and delete them all except the module objects and the private objects that starts or ends with the "__" string.

If we don't do this, then the second run can use a variable defined from the first run.

Example:

PyRun_SimpleStringFlags("print test", NULL); // Error: test variable not defined.
...
PyRun_SimpleStringFlags("test = 12", NULL); // Defines the test variable.
...
PyRun_SimpleStringFlags("print test", NULL); // Prints 12 since the variable is still defined from the previous run because we haven't called Py_Finalize().

Sorry if my work around example contains non-included functions. The StartsWith() and EndsWith() functions simply checks if the string starts with __ or ends with __. The boost::starts_with() and boost::ends_with() functions could be used instead. The GetUnicodeString() function is simply a function that returns a UTF8 string, it uses PyUnicode_AsWideChar() or PyString_AsString() depending on the object type.

To create the main module, I use:
PyObject * poMainModule = PyImport_AddModule("__main__");
Regards,

Phelippe

Thank you for your information, Phelippe.
I hope that numpy developers finally will take a look and solve this issue.
In the meantime a good workaround like this can be tried and applied.

One of the problem with the work around is that if a module is imported on the first run, it will not be imported on subsequent runs. So if the user imports a in-house script, but changes it to fix an issue, he would need to close the your application and start it again for his changes to take effect.

@phelippeneveu

@phelippeneveu Yes, but with importlib.relaod() he can do the job (with some hack in main script ?)

EDIT : error in function name, sorry =)

I am struggling with the same issue. It does not just apply to numpy...
One issue is that Python itself does not unload the module DLLs on PyFinalize(). Unlike , say Matlab, where 'clear mex' will actually unload any loaded MEX DLL's. PyFinalize should unload all the .pyd (DLLs). That would truly set the modules back to initial state when they are loaded again.
Just to clarify.
Assume your DLL (.pyd) has some 'C' code with a variable
static int myImportantInitialCounter=0;
And inside the DLL that variable is incremented and used.
After a PyFinalize(), that value will remain at it's incremented value, even after a reload of the python module as the DLL module is still loaded in memory.
If PyFinalize properly unloaded all loaded DLL modules then, after a PyInitialize, the next time the python module was loaded the variable would be back to 0 when the DLL module is loaded and initialized.

@loopless

@loopless If you want to propose changes to how Python handles extension modules then you should talk to the Python team, not us :-)

@njsmith

@njsmith I think it is useful for people to understand what is going on under the hood, even if it off-topic, so that they realize a resolution is not going to come from the numpy team.

Same issue here. It is quite disappointing for me that this is not working, and so there really is no good solution to re-initialize the embedded python interpreter for scripts using numpy.

The use case here is an application which embeds a python interpreter for scripting vision algorithms. Unfortunately, this bug forces us to stop and start the application each time we make changes to the scripts. I know there is importlib.reload, but it is not really usable either for complex module dependencies.

Note: Without numpy, the Initialize - Finalize works great and we don't need to restart after script editing.

I don't know, is there maybe a good solution out there using sockets and seperate processes? It looks like this problem must be hitting a lot of people (i.e., everyone who wants to use embedded python together with numpy).

Just adding to the general cry-out (and raise impotance level of the issue). We are experiencing exactly the same issues and as a Python interpreter is not essential in our application, there is discussion of dropping the project alltogether if it isn't properly solved soon. This would be a real shame and a great opportunity lost.

I don't know much about this topic and don't know any developers who embed Numpy. It would help if someone more knowledgeable could explain why embedding is a problem.

Unfortunately this is a fundamental problem with Python and a huge PITA for anyone embedding Python as a simple "restart interpreter" seems difficult if not impossible to get to function correctly.

But as noted previously, requests should be directed at the Python developers. In my opinion, Py_Finalize() should unload all DLLs (.pyd) that Python has loaded. This almost certainly solve the problem.

PS On Windows at least, DLL Handles have a reference count so that if the host app also loaded a DLL, it won't get unloaded from behind your back.

I've just done that: https://bugs.python.org/issue34309

Having said that, I still believe that there are problems in numpy cleanup-code as well.

We experienced this same issue with numpy in embedded python interpreter. Our workaround was simply to not use numpy at all (there is no easy way for us to avoid re-initialisation of the interpreter, and an exit and reload of the app each time is not practical). It was disappointing to not be able to use numpy, as it meant we had to hand-roll all the stuff that we wanted numpy to do for us.

[Python 3] fix python_parser test

Navigation Menu

Repository navigation

Crash when importing numpy from the Python C-API after calling Py_Finilize() #8097

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Crash when importing numpy from the Python C-API after calling Py_Finilize() #8097

Description

Activity

natebazar commented on May 3, 2017

njsmith commented on May 3, 2017

den-run-ai commented on Jan 10, 2018

pv commented on Jan 10, 2018

phelippeneveu commented on Jan 10, 2018

gyp33 commented on Jan 12, 2018

AndreaLazz commented on Apr 18, 2018

Takishima commented on Apr 18, 2018

phelippeneveu commented on Apr 18, 2018

AndreaLazz commented on Apr 19, 2018

phelippeneveu commented on Apr 19, 2018

Archloy commented on Apr 19, 2018

loopless commented on Jun 28, 2018

njsmith commented on Jun 29, 2018

loopless commented on Jun 29, 2018

wiedemannc commented on Jul 31, 2018

Bejoscha commented on Aug 1, 2018

charris commented on Aug 1, 2018

loopless commented on Aug 1, 2018

wiedemannc commented on Aug 1, 2018

0xworks commented on Nov 6, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions