Deadlock during the "import" in the fork()'ed child process if fork() happened while import_lock was held #50629

Activity

1. hdn commented on Jun 29, 2009

   

   hdnmannequin

   on Jun 29, 2009

   MannequinAuthor

   More actions

   Greetings,

   The 2.4 and 2.5 versions of python contains a deadlock caused by
   possibility to hold import_lock while doing fork() and not resetting it
   in the child (on the linux platform).

   The prove of concept code is:

   --BEGIN (import_lock.py)--

   #!/usr/bin/python2.4

   import os
   import time
   import threading
   
   class SecondThread(threading.Thread):
     def run(self):
       # Give the main thread time to hold import_lock and start importing.
       time.sleep(1)
   
       # Fork the process while holding import_lock in the main thread.
       pid = os.fork()
   
       if pid == 0:  # Child process
         print "child begin"

     \# The import lock is still taken by main thread which is now not
   

   the part
   # of the child process. The import lock will never be released in the
   # child process. Effectively, any import is a deadlock from now on.
   import types

     # This statement will never be executed.
     print "child end"
   

   def main():
     second_thread = SecondThread()
     second_thread.start()
   
     # Take the import_lock and then release global interpreter lock in the
     # import_lock_helper module by calling any blocking operation.
     import import_lock_helper
   
     second_thread.join()
   
   main()

   --END (import_lock.py)--

   --BEGIN (import_lock_helper.py)--

   #!/usr/bin/python2.4

   import time
   
   # Release the global interpreter lock by calling any blocking operation.
   time.sleep(10)

   --END (import_lock_helper.py)--

   The stack of the child python interpreter at the time of dead lock:

   (gdb) bt
   #0 0xffffe410 in __kernel_vsyscall ()
   #1 0xf7f81700 in sem_wait@GLIBC_2.0 () from
   /usr/grte/v1/lib/libpthread.so.0
   #2 0x081ab500 in ?? ()
   #3 0x080e1855 in PyThread_acquire_lock (lock=0x0, waitflag=1) at
   ../../Python/thread_pthread.h:313
   #4 0x080d1f3b in lock_import () at ../../Python/import.c:247
   #5 0x080d52a4 in PyImport_ImportModuleEx (name=0xf7e0f8f4 "types",
   globals=0xf7def824, locals=0x8123cb8, fromlist=0x8123cb8) at
   ../../Python/import.c:1976
   #6 0x080af2d0 in builtin___import__ (self=0x0, args=0xf7db7cd4) at
   ../../Python/bltinmodule.c:45
   #7 0x08058d77 in PyObject_Call (func=0x0, arg=0xf7db7cd4, kw=0x0) at
   ../../Objects/abstract.c:1795
   #8 0x080b30ec in PyEval_CallObjectWithKeywords (func=0xf7ddfd6c,
   arg=0xf7db7cd4, kw=0x0) at ../../Python/ceval.c:3435
   #9 0x080b5ca6 in PyEval_EvalFrame (f=0x8167a04) at
   ../../Python/ceval.c:2020
   #10 0x080b942c in PyEval_EvalFrame (f=0x81ab57c) at
   ../../Python/ceval.c:3651
   . . . .

   (gdb) pystack
   import_lock.py (26): run
   /usr/lib/python2.4/threading.py (443): __bootstrap

   The code directly responsible for import locking (Python/import.c):

   --BEGIN--
   static PyThread_type_lock import_lock = 0;
   static long import_lock_thread = -1;
   static int import_lock_level = 0;
   
   static void
   lock_import(void)
   {
          long me = PyThread_get_thread_ident();
          if (me == -1)
                  return; /* Too bad */
          if (import_lock == NULL) {
                  import_lock = PyThread_allocate_lock();
                  if (import_lock == NULL)
                          return;  /* Nothing much we can do. */
          }
          if (import_lock_thread == me) {
                  import_lock_level++;
                  return;
          }
          if (import_lock_thread != -1 ||
   !PyThread_acquire_lock(import_lock, 0))
          {
                  PyThreadState *tstate = PyEval_SaveThread();
                  PyThread_acquire_lock(import_lock, 1);
                  PyEval_RestoreThread(tstate);
          }
          import_lock_thread = me;
          import_lock_level = 1;
   }
   
   static int
   unlock_import(void)
   {
          long me = PyThread_get_thread_ident();
          if (me == -1 || import_lock == NULL)
                  return 0; /* Too bad */
          if (import_lock_thread != me)
                  return -1;
          import_lock_level--;
          if (import_lock_level == 0) {
                  import_lock_thread = -1;
                  PyThread_release_lock(import_lock);
          }
          return 1;
   }

   /* This function is called from PyOS_AfterFork to ensure that newly
   created child processes do not share locks with the parent. */

   void
   _PyImport_ReInitLock(void)
   {
   #ifdef _AIX
          if (import_lock != NULL)
                  import_lock = PyThread_allocate_lock();
   #endif
   }
   --END--

   The possible solution is to reset import_lock in the
   _PyImport_ReInitLock() not only for _AIX but also for Linux and maybe
   other platforms (do you know why _AIX-only guard is there?).

   --CUT HERE--
   void
   _PyImport_ReInitLock(void)
   {
          if (import_lock != NULL)
                  import_lock = PyThread_allocate_lock();
   }
   --CUT HERE--

   Prove of concept example above works fine (w/o deadlocks) on the python
   interpreter rebuilt with the _PyImport_ReInitLock() modification above.

   Also this bug can be worked around in Python code by holding import_lock
   before fork() and releasing import_lock right after fork() in both
   parent and child.

   The workaround code is:

   --BEGIN (workaround_fork_import_bug.py)--

   import imp
   import os
   
   def __fork():
     imp.acquire_lock()
     try:
       return _os_fork()
     finally:
       imp.release_lock()

   try:
   _os_fork
   except NameError:
   _os_fork = os.fork
   os.fork = __fork

   --END (workaround_fork_import_bug.py)--

   This workaround can also be implemented in Python interpreter in C and
   could be other solution for this bug.

   Thanks,
   Dmitriy

   $ uname -srvmpio
   Linux 2.6.24-gg24-generic #1 SMP Wed Apr 22 21:48:06 PDT 2009 x86_64
   unknown unknown GNU/Linux

   P.S. The problem described above is probably causes (some) effects
   described in http://bugs.python.org/issue1590864.
2. hdn

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jun 29, 2009
3. loewis commented on Jun 30, 2009

   

   loewismannequin

   on Jun 30, 2009

   Mannequin

   More actions

   Does the problem also exist in Python 2.6? We will definitely not fix it
   anymore for 2.4 and 2.5.
4. hdn commented on Jun 30, 2009

   

   hdnmannequin

   on Jun 30, 2009

   MannequinAuthor

   More actions

   Does the problem also exist in Python 2.6? We will definitely not fix it
   anymore for 2.4 and 2.5.

   Yep. Exactly same problem in Python 2.6.

   This problem does probably exist in all newer versions as well but I
   didn't explicitly test for that.
5. gpshead commented on Apr 14, 2012

   

   gpshead

   on Apr 14, 2012

   Member

   More actions

   What is the status of this in 2.7?

   Brett - what about in 3.3 after you get importlib in?
6. gpshead commented on Apr 14, 2012

   

   gpshead

   on Apr 14, 2012

   Member

   More actions

   btw, a potentially related (or duplicate?) issue was already fixed - http://bugs.python.org/issue1590864
7. pitrou commented on Apr 14, 2012

   

   pitrou

   on Apr 14, 2012

   Member

   More actions

   This was fixed long ago in 724bbd489ad4. Dmitriy's example works fine with 2.7.
8. pitrou

   closed this as completedon Apr 14, 2012
9. brettcannon commented on Apr 14, 2012

   

   brettcannon

   on Apr 14, 2012

   Member

   More actions

   Just to answer Greg's question, importlib uses a context manager to manage the import lock so as long as that surfaces the right thing in a fork then things will be fine.
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022