subprocess wait() hangs when stdin is closed #56995

Activity

1. idank commented on Aug 19, 2011

   

   idankmannequin

   on Aug 19, 2011

   MannequinAuthor

   More actions

   The following program hangs on Debian, Python 2.6.6:

   import subprocess
   
   proc1 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   proc2 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   
   proc1.stdin.close()
   proc1.wait()

   Changing the last two lines to:

   proc2.stdin.close()
   proc2.wait()

   Doesn't hang. The guys at #python-dev confirmed the same happens on 2.7 but not on 3.x.
2. idank

   added

   performancePerformance or resource usagePerformance or resource usage

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Aug 19, 2011
3. pitrou

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   and removed

   performancePerformance or resource usagePerformance or resource usage

   on Aug 19, 2011
4. BenWolfson commented on Aug 19, 2011

   

   BenWolfsonmannequin

   on Aug 19, 2011

   Mannequin

   More actions

   "The guys at #python-dev confirmed the same happens on 2.7 but not on 3.x."

   Really? This is on gentoo, not debian, admittedly:

   coelacanth ~ 11:12:36 $ python3
   Python 3.1.3 (r313:86834, May  1 2011, 09:41:48) 
   [GCC 4.4.4] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import subprocess
   >>> proc1 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   >>> proc2 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   >>> proc2.stdin.close()
   >>> proc2.wait()
   0
   >>> 
   
   
   
   coelacanth ~ 11:12:13 $ python3.1
   Python 3.1.3 (r313:86834, May  1 2011, 09:41:48) 
   [GCC 4.4.4] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import subprocess
   >>> proc1 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   >>> proc2 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   >>> proc1.stdin.close()
   >>> proc1.wait()

   [hangs]
5. neologix commented on Aug 19, 2011

   

   neologixmannequin

   on Aug 19, 2011

   Mannequin

   More actions

   Hello Idan,

   The following program hangs on Debian

   Debian is a good choice :-)

   Concerning your example, it hangs is because the file descriptor used
   to communicate with proc1 is inherited when proc2 is created (FDs are
   inherited upon fork by default): thus, when you call
   proc1.stdin.close(), cat doesn't receive EOF on read(0), and remains
   stuck.
   If you close proc2's stdin and wait it first, then you don't have this problem:

   • because it's been created after proc1, its stdin FD has not been inherited
   • when you call proc2.stdin.close(), cat receives EOF from read(0), and exits

   There are two reasons while it doesn't hang on Python 3.x:

   1. it uses close_fds=True by default
   2. it sets the pipe FDs CLOEXEC, so that they're closed when cat is executed

   Try with
   """
   proc1 = subprocess.Popen(['cat'], stdin=subprocess.PIPE)
   proc2 = subprocess.Popen(['cat'], stdin=subprocess.PIPE, close_fds=True)

   proc1.stdin.close()
   proc1.wait()
   """

   And you'll be fine.

   We can't make the change 1) in 2.7, because that's a behavior change.
   We could however set the FDs used to communicate with the children CLOEXEC.
   I'll work on a patch for 2.7 (but it won't be thread-safe, because
   pipe2 is not exposed and there's no _posixsubprocess.c like in 3.x).
6. neologix commented on Aug 19, 2011

   

   neologixmannequin

   on Aug 19, 2011

   Mannequin

   More actions

   Here's a patch + test for 2.7.

   Really? This is on gentoo, not debian, admittedly:

   That's because the change of close_fds to True by default and the CLOEXEC flag were added in 3.2.
   Since 3.1 is in security-fix mode only, this patch is only relevant to 2.7.
7. idank commented on Aug 20, 2011

   

   idankmannequin

   on Aug 20, 2011

   MannequinAuthor

   More actions

   Thanks for getting on top of this so quickly Charles. Setting close_fds=True worked like a charm.
8. pitrou commented on Aug 24, 2011

   

   pitrou

   on Aug 24, 2011

   Member

   More actions

   You might want to add a comment in the patch that the cloexec flag is removed from the child's pipes by calling dup2() before exec() :) I was a bit baffled at first when reading the patch.
9. neologix commented on Aug 24, 2011

   

   neologixmannequin

   on Aug 24, 2011

   Mannequin

   More actions

   With comment.
   I'll add a similar comment to default.
10. vstinner commented on Aug 24, 2011

    

    vstinner

    on Aug 24, 2011

    Member

    More actions

    subprocess_cloexec-1.diff: I'm too tired too review the test. The subprocess.py part looks correct, except the pipe2 name. Python 3 uses "_create_pipe = _posixsubprocess.cloexec_pipe". Pick one of those.
11. rosslagerwall commented on Aug 25, 2011

    

    rosslagerwallmannequin

    on Aug 25, 2011

    Mannequin

    More actions

    Patch looks good after Victor's comment.
12. python-dev commented on Aug 25, 2011

    

    python-devmannequin

    on Aug 25, 2011

    Mannequin

    More actions

    New changeset 498b03a55297 by Charles-François Natali in branch '2.7':
    Issue bpo-12786: Set communication pipes used by subprocess.Popen CLOEXEC to avoid
    http://hg.python.org/cpython/rev/498b03a55297
13. neologix commented on Aug 25, 2011

    

    neologixmannequin

    on Aug 25, 2011

    Mannequin

    More actions

    Patch committed.
    Idan, thanks for the report.
14. neologix

    closed this as completedon Aug 25, 2011
15. ezio-melotti

    transferred this issue fromon Apr 10, 2022