Repository navigation
Segfault in urllib.requests.urlopen and/or ThreadPoolExecutor #10008
Description
Description
Trying to test the cache mechanism by hammering it with multiple threads seems to result in segmentation faults.
May be related to #9699
Expected behavior
Tests pass, or fail in a way that pytest can catch.
Actual behavior
Segfault occurs in, for example, test_download_file_threaded_many.
Steps to Reproduce
def test_threaded_segfault(valid_urls):
"""Demonstrate urllib's segfault."""
def slurp_url(u):
with urllib.request.urlopen(u) as remote:
block = True
while block:
block = remote.read(1024)
urls = list(islice(valid_urls, N_THREAD_HAMMER))
with ThreadPoolExecutor(max_workers=len(urls)) as P:
list(P.map(lambda u: slurp_url(u),
[u for (u, c) in urls]))You will notice that no actual astropy function is called here; the problem is that the stock astropy test suite triggers this segfault. #10007 sets all such tests to be skipped.
System Details
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; print(platform.platform())
Linux-5.3.0-29-generic-x86_64-with-Ubuntu-19.10-eoan
>>> import sys; print("Python", sys.version)
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
[GCC 8.3.0]
>>> import numpy; print("Numpy", numpy.__version__)
Numpy 1.17.1
>>> import astropy; print("astropy", astropy.__version__)
astropy 4.1.dev615+g4f43a53f6
>>> import scipy; print("Scipy", scipy.__version__)
Scipy 1.3.1
Activity
aarchiba commentedon Mar 4, 2020
Someone should do the exploration needed to nail down the bug so it can be filed against upstream. I realize that this kind of statement should end with an implicit "... and I volunteer" but I'm a little swamped just now.
aarchiba commentedon Mar 4, 2020
Okay it goes away with python 3.8. Here's a completely standalone demo:
pllim commentedon Mar 4, 2020
OK, so they fixed it upstream. Shall we close this?
aarchiba commentedon Mar 4, 2020
Well it still causes crashes on 3.7, at least the version on my Ubuntu laptop. But I guess as long as we don't re-enable those tests people won't start running into it.
aarchiba commentedon Mar 4, 2020
Upstream issue https://bugs.python.org/issue39853
dhomeier commentedon Mar 5, 2020
I tried to reproduce the segfaults with your standalone demo, but could not trigger any on OS X 10.12.6 with either 3.6.10 or 3.7.6 a couple 100 times. The test failures as reported in #9699 did occur with both Python versions, but strangely even they are no longer as reproducible:
python3.N -c 'import astropy; astropy.test(package="utils", args="-vk \" _open_network_url or _download\"", remote_data=True)'reliably raises the
Current thread 0x00007fffd33d43c0 (most recent call first): File "/sw/lib/python3.6/socket.py", line 745 in getaddrinfoexception whenever tested under 3.6.10
With 3.7.6 it consistently still comes up in any astropy installation built in my own user space, but never with the one built in the closed Fink dpkg build environment, which is essentially using for Astropy 4.0
python3.7 setup.py build --offline --use-system-cfitsio --use-system-expat python3.7 setup.py install --offline --root=/sw- orPYTHONPATH=/opt/lib/python3.7/site-packages python3.7 setup.py install --offline --prefix=/optfor an alternative test. This is building as a non-privileged user and installing as root, but otherwise using the same compilers, and no relevant environment differences I could think of.
Anyway I probably would not label this as 'fixed upstream' until the real issue is understood, and possibly a fix backported to 3.6 and 3.7, at least as long as 3.6 is officially supported.
bsipocz commentedon Mar 5, 2020
OK, let's then keep it open.
mhvk commentedon Oct 22, 2020
Also seems to be a problem on Windows CI (see #10914)
pllim commentedon Oct 22, 2020
Re: #10008 (comment)
@mhvk , the Windows case ( #10914) is not a segfault, but more like some race condition resulting in failure to delete a cache, which then causes failure in test tear-down (as far as I can understand by reading the traceback).