Skip to content

MachAr._do_init() hangs under mod_wsgi #3837

New issue
New issue
Closed
Closed
MachAr._do_init() hangs under mod_wsgi#3837
Labels
00 - BugEmbeddedIssues regarding embedded python interpreterscomponent: numpy._core

Description

@lanzz
lanzz
opened on Sep 30, 2013

(numpy version 1.6.2)

I've been investigating a complete process hang during from sklearn.cluster import DBSCAN, and I've traced it to a call to numpy.finfo(float) in the scipy.linalg.decomp_schur module. The actual hang occurs in core/machar.py line 227:

temp = z*t

This line is within a for loop and occurs on the _ = 9 iteration. At that point I'm getting repr(z) = array([ 5.56268465e-309]), and repr(t) = array([ 1.]) (I don't know how to get a more accurate representation of their actual state). The next line is never reached — the process hangs indefinitely at the temp = z*t line.

The weirdest part is that the from sklearn.cluster import DBSCAN import works fine when I try it in a clean Python shell, but hangs when running in our web application. We're using numpy very sparsely, and we're certainly not messing with its internals, so I have no idea how to narrow down the conflict that apparently occurs between numpy and the rest of our codebase.

Activity

  1. juliantaylor commented on Sep 30, 2013

    @juliantaylor
    juliantaylor
    on Sep 30, 2013
    Contributor

    what kind of hardware are you using? np.finfo has issues on some machines, see e.g. #3125
    can you produce a small testcase?

  2. juliantaylor commented on Sep 30, 2013

    @juliantaylor
    juliantaylor
    on Sep 30, 2013
    Contributor

    that its hanging on z*t is very strange, can you debug into this with gdb?

    gdb --args <your program> <arguments>
run
<wait until hang, presse Ctrl+C>
backtrace full

  3. lanzz commented on Oct 1, 2013

    @lanzz
    lanzz
    on Oct 1, 2013
    Author

    I'm running on a i686 Celeron, 32-bit Ubuntu installation. I think that's as vanilla as it can be.
    The issue occurs in a WSGI web application running under Apache, I'm not really sure how to approach debugging it within Apache. Running it as a Python webserver does not trigger the issue, numpy.finfo(float) returns successfully.

    My testcase is simply numpy.finfo(float). Hangs when running in Apache, works in the Python shell and as a standalone WSGI application.

  4. lanzz commented on Oct 1, 2013

    @lanzz
    lanzz
    on Oct 1, 2013
    Author

    Minimal WSGI application that reproduces the issue when executed under mod_wsgi:

    import numpy

def application(*args, **kwargs):
    finfo = numpy.finfo(float)
    raise Exception(finfo)

    Exception is never raised, request hangs without response.

  5. pv commented on Oct 1, 2013

    @pv
    pv
    on Oct 1, 2013
    Member

    Which Ubuntu version, and which apache/mod_wsgi package versions?

  6. pv commented on Oct 1, 2013

    @pv
    pv
    on Oct 1, 2013
    Member

    Googling "numpy mod_wsgi": http://serverfault.com/questions/514242/non-responsive-apache-mod-wsgi-after-installing-scipy

    http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

  7. pv commented on Oct 1, 2013

    @pv
    pv
    on Oct 1, 2013
    Member

    The issue is using PyGILState_Ensure, which Python docs say is not compatible with sub-interpreters (which mod_wsgi uses in its default configuration).

  8. njsmith commented on Oct 1, 2013

    @njsmith
    njsmith
    on Oct 1, 2013
    Member

    That sounds like a pretty plausible explanation.

    @original filer: Does the WSGIApplicationGroup %{GLOBAL} hack work?

    Anyone know if there's something we can/should do to fix this on our end?
    The mod_wsgi docs are not very helpful to library writers.
    On 1 Oct 2013 09:47, "Pauli Virtanen" [email protected] wrote:

    Googling:
    http://serverfault.com/questions/514242/non-responsive-apache-mod-wsgi-after-installing-scipy

    http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/3837#issuecomment-25433867
    .

  9. pv commented on Oct 1, 2013

    @pv
    pv
    on Oct 1, 2013
    Member

    @njsmith: as far as I understand, replacing calls to PyGILState_Ensure with the other equivalent (?) Python mechanisms. However, also Cython uses PyGILState_Ensure, so it would not be enough to "fix" Numpy.

    Python sub-interpreters probably don't mix well with any extension modules.

  10. njsmith commented on Oct 1, 2013

    @njsmith
    njsmith
    on Oct 1, 2013
    Member

    Gotcha. Well, sounds like something that if someone feels inspired to fix
    then that'd be cool, but not really urgent given how many libraries have
    this problem with mod_wsgi and that there are well documented workarounds.
    On 1 Oct 2013 10:16, "Pauli Virtanen" [email protected] wrote:

    @njsmith https://github.com/njsmith: as far as I understand, replacing
    calls to PyGILState_Ensure with the other equivalent (?) Python
    mechanisms. However, also Cython uses PyGILState_Ensure, so it would not
    be enough to "fix" Numpy.

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/3837#issuecomment-25435316
    .

  11. lanzz commented on Oct 1, 2013

    @lanzz
    lanzz
    on Oct 1, 2013
    Author

    Yes, the WSGIApplicationGroup %{GLOBAL} hack works. Thanks for the help, I wasn't able to narrow my searches enough to find that SO question.

    In my case this is an acceptable workaround, probably it would be good to have it prominently documented somewhere?

  12. ndjensen commented on May 8, 2015

    @ndjensen
    ndjensen
    on May 8, 2015
    Contributor

    This is definitely not a trivial thing to solve, but if you wanted to replace PyGILState_Ensure() then I think you can as long as you have the PyThreadState*. Getting that could be tricky, Jep uses the PyThreadState_GetDict() method. See https://github.com/mrj0/jep/blob/v3.3.0rc/src/jep/pyembed.c#L351

    PyEval_AcquireLock();
PyThreadState *prevThread = PyThreadState_Swap(tstate);
// do stuff
PyThreadState_Swap(prevThread);
PyEval_RelaseLock();

    For an example of Jep doing this, see https://github.com/mrj0/jep/blob/v3.3.0rc/src/jep/pyembed.c#L708

  13. charris
    added
    EmbeddedIssues regarding embedded python interpreters
    on May 9, 2015

  14. elliottslaughter commented on Jun 25, 2018

    @elliottslaughter
    elliottslaughter
    on Jun 25, 2018

    I am experiencing this freeze (on the exact same line of code) on Numpy 1.11.3 in an application that does not use wsgi (it's not a web app) or sub-interpreters. The code runs fine on my desktop but freezes deterministically when I run it on the Cori supercomputer, which runs an Intel KNL architecture.

    Is there any good way to determine what components might be responsible? At the moment I don't have any good ways to debug this.

  15. charris commented on Jun 25, 2018

    @charris
    charris
    on Jun 25, 2018
    Member

    The code is still there in current NumPy, but has been superseded and should not be accessed in normal use from finfo if you have a standard floating type. If it gets fixed in the future, it will preferably be by removal. Could you try a more recent NumPy and see if that fixes the problem?

  16. elliottslaughter commented on Jun 27, 2018

    @elliottslaughter
    elliottslaughter
    on Jun 27, 2018

    I upgraded to NumPy 1.13.3 and the problem went away. Thank you!

  17. charris
    closed this as completedon Jun 27, 2018

  18. charris commented on Jun 27, 2018

    @charris
    charris
    on Jun 27, 2018
    Member

    Thanks for checking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    00 - BugEmbeddedIssues regarding embedded python interpreterscomponent: numpy._core

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions