Skip to content

Segmentation fault with urllib.request.urlopen and threads #84034

New issue
New issue
Closed
Closed
Segmentation fault with urllib.request.urlopen and threads#84034
Labels
3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-crashA hard crash of the interpreter, possibly with a core dump

Description

@AnneArchibald
AnneArchibald
mannequin
opened on Mar 4, 2020
BPO 39853
Nosy @vstinner, @Zheaoli, @gilles-duboscq
Files
  • urllib_segfault.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-03-26.00:24:53.447>
created_at = <Date 2020-03-04.18:38:22.960>
labels = ['3.7', 'library', 'type-crash']
title = 'Segmentation fault with urllib.request.urlopen and threads'
updated_at = <Date 2021-02-18.18:19:35.745>
user = 'https://bugs.python.org/AnneArchibald'

    bugs.python.org fields:

    activity = <Date 2021-02-18.18:19:35.745>
actor = 'gilles-duboscq'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-26.00:24:53.447>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-03-04.18:38:22.960>
creator = 'Anne Archibald'
dependencies = []
files = ['48950']
hgrepos = []
issue_num = 39853
keywords = []
message_count = 13.0
messages = ['363374', '364873', '364876', '364881', '364882', '364890', '365035', '387246', '387249', '387250', '387253', '387254', '387259']
nosy_count = 4.0
nosy_names = ['vstinner', 'Manjusaka', 'Anne Archibald', 'gilles-duboscq']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39853'
versions = ['Python 3.7']

    Activity

    1. AnneArchibald commented on Mar 4, 2020

      @AnneArchibald
      AnneArchibaldmannequin
      on Mar 4, 2020
      MannequinAuthor

      This was discovered in the astropy test suite, where ThreadPoolExecutor is used to concurrently launch a lot of urllib.request.urlopen. This occurs when the URLs are local files; I'm not sure about other URL schemes.

      The problem appears to occur in python 3.7 but not python 3.8 or python 3.6 (on a different machine).

      $ python urllib_segfault.py
Linux-5.3.0-29-generic-x86_64-with-Ubuntu-19.10-eoan
Python 3.7.3 (default, Apr  3 2019, 05:39:12) 
[GCC 8.3.0]
Segmentation fault (core dumped)
$ python3.8 urllib_segfault.py
Linux-5.3.0-29-generic-x86_64-with-glibc2.29
Python 3.8.0 (default, Oct 28 2019, 16:14:01) 
[GCC 9.2.1 20191008]
$ python3 urllib_segfault.py 
Linux-4.15.0-88-generic-x86_64-with-Ubuntu-18.04-bionic
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]
$

      The Astropy bug report: astropy/astropy#10008

    2. AnneArchibald
      added
      3.7 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Mar 4, 2020

    3. Zheaoli commented on Mar 23, 2020

      @Zheaoli
      Zheaolimannequin
      on Mar 23, 2020
      Mannequin

      I have tried it on Python 3.7.3 Ubuntu 18.04

      Linux-4.15.0-1060-aws-x86_64-with-debian-buster-sid
      Python 3.7.3 (default, Mar 23 2020, 18:15:26)
      [GCC 7.5.0]

      there is no segmentation fault

      would you mind sharing the core dump to help us find more detail about the crash?

    4. Zheaoli commented on Mar 23, 2020

      @Zheaoli
      Zheaolimannequin
      on Mar 23, 2020
      Mannequin

      use the same GCC 8.3.0 to recompile the Python 3.7.3 still no fault

      maybe we need the core dump to figure it out.

    5. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      Python 3.7.3 is outdated. Can you please try on newer Python version?

      I failed to reproduce the crash on Fedora 31. I tested:

      vstinner@apu$ python3.7 -VV
      Python 3.7.6 (default, Jan 30 2020, 09:44:41)
      [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

      vstinner@apu$ python3.8 -VV
      Python 3.8.2 (default, Feb 26 2020, 00:00:00)
      [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

      vstinner@apu$ python3.9 -VV
      Python 3.9.0a4 (default, Feb 27 2020, 00:00:00)
      [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

      vstinner@apu$ python3-debug -VV
      Python 3.7.6 (default, Jan 30 2020, 09:04:19)
      [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

    6. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      Try to get the gdb traceback (C call stack).

      You may also try: python3 -X faulthandler urllib_segfault.py.

      But I expect a crash in Python finalization, where there is no Python frame and so empty traceback.

    7. Zheaoli commented on Mar 23, 2020

      @Zheaoli
      Zheaolimannequin
      on Mar 23, 2020
      Mannequin

      Hello Victor

      I have tried on both MacOS and Ubuntu 18.04 from 3.8.2 to the newest code in master, and can't reproduce this problem

      macOS-10.15.4-x86_64-i386-64bit
      Python 3.9.0a4+ (heads/master:9a81ab107a, Mar 24 2020, 02:06:30)
      [Clang 11.0.3 (clang-1103.0.32.26)]

      Linux-4.15.0-1060-aws-x86_64-with-glibc2.27
      Python 3.8.2 (default, Mar 23 2020, 19:21:16)
      [GCC 8.3.0]

      Linux-4.15.0-1060-aws-x86_64-with-glibc2.27
      Python 3.9.0a4 (default, Mar 23 2020, 19:25:43)
      [GCC 8.3.0]

      I think it may be caused by the kernel version and the glibc version, I don't have enough evidence to prove it.

      I will try the Python 3.7.6 to the newest version in Ubuntu 19.10 with Kernel 5.3.0-29 and GCC 8.3.0

    8. vstinner commented on Mar 26, 2020

      @vstinner
      vstinner
      on Mar 26, 2020
      Member

      If someone manages to reproduce the bug, please provide *at least* the Python traceback. You can use faulthandler to get it. It would also be very useful to get the C call stack using gdb (where command in gdb).

      In the meanwhile, I close the issue since only the reporter reproduced the crashed.

      Please provide more information about your OS, OS version, how you installed Python, Python package version, etc. if you want to reopen the issue.

    9. vstinner
      closed this as completedon Mar 26, 2020
    10. vstinner
      closed this as completedon Mar 26, 2020

    11. gilles-duboscq commented on Feb 18, 2021

      @gilles-duboscq
      gilles-duboscqmannequin
      on Feb 18, 2021
      Mannequin

      I'm not sure it's the same but we have seen stack traces looking like the one there: astropy/astropy#9699

      Current thread 0x00007fffa857e3c0 (most recent call first):
      File "/sw/lib/python3.7/socket.py", line 748 in getaddrinfo
      File "/sw/lib/python3.7/socket.py", line 707 in create_connection
      File "/sw/lib/python3.7/http/client.py", line 938 in connect
      File "/sw/lib/python3.7/http/client.py", line 966 in send
      File "/sw/lib/python3.7/http/client.py", line 1026 in _send_output
      File "/sw/lib/python3.7/http/client.py", line 1247 in endheaders
      File "/sw/lib/python3.7/http/client.py", line 1298 in _send_request
      File "/sw/lib/python3.7/http/client.py", line 1252 in request

      For us it happens when using multiprocessing: the main process forks 2 processes and both use urlopen at roughly the same time. We are seeing this on Python 3.7.2 on macOS 10.14.3.

    12. vstinner commented on Feb 18, 2021

      @vstinner
      vstinner
      on Feb 18, 2021
      Member

      "Segmentation fault with (...) threads (...) getaddrinfo"

      Aha, another victim on a getaddrinfo() implementation which is not thread safe.

      See this code in Modules/socketmodule.c:

      /* Lock to allow python interpreter to continue, but only allow one
   thread to be in gethostbyname or getaddrinfo */
#if defined(USE_GETHOSTBYNAME_LOCK)
static PyThread_type_lock netdb_lock;
#endif

      Can you please check if your Python was built with HAVE_GETHOSTBYNAME_R?

      $ python3
Python 3.9.1 (default, Jan 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
>>> import sysconfig; repr(sysconfig.get_config_var('HAVE_GETHOSTBYNAME_R'))
'1'

      Modules/socketmodule.c is full of #ifdef involving macOS...

    13. gilles-duboscq commented on Feb 18, 2021

      @gilles-duboscq
      gilles-duboscqmannequin
      on Feb 18, 2021
      Mannequin

      I get '0' so it was not built with HAVE_GETHOSTBYNAME_R.

    14. vstinner commented on Feb 18, 2021

      @vstinner
      vstinner
      on Feb 18, 2021
      Member

      Oh wait, I removed that lock:

      commit 0de437d
      Author: Victor Stinner <[email protected]>
      Date: Thu May 28 17:23:39 2020 +0200

      bpo-25920: Remove socket.getaddrinfo() lock on macOS (GH-20177)

On macOS, socket.getaddrinfo() no longer uses an internal lock to
prevent race conditions when calling getaddrinfo(). getaddrinfo is
thread-safe is macOS 10.5, whereas Python 3.9 requires macOS 10.6 or
newer.

The lock was also used on FreeBSD older than 5.3, OpenBSD older than
201311 and NetBSD older than 4.

      Please open a new issue specific to macOS: specify your macOS version, your Python version and how you installed or built Python.

    15. vstinner commented on Feb 18, 2021

      @vstinner
      vstinner
      on Feb 18, 2021
      Member

      For us it happens when using multiprocessing: the main process forks 2 processes and both use urlopen at roughly the same time. We are seeing this on Python 3.7.2 on macOS 10.14.3.

      From what I understood, calling fork() (and then continue to execute regular Python code) is no longer safe in macOS 10.14 and must no longer be used. Only fork+exec is safe (spawn a new child process), but posix_spawn() is preferred on macOS. But I'm not a macOS expert.

    16. gilles-duboscq commented on Feb 18, 2021

      @gilles-duboscq
      gilles-duboscqmannequin
      on Feb 18, 2021
      Mannequin

      Thanks Victor we'll look into moving away from this pattern.

    17. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions