Repository navigation
test_semaphore_tracker() of test_multiprocessing_spawn fails randomly (race condition?) #75868
Description
| BPO | 31687 |
|---|---|
| Nosy | @ncoghlan, @vstinner, @Subler |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-09-19.23:27:14.912>
created_at = <Date 2017-10-04.09:36:35.852>
labels = ['3.7', 'tests']
title = 'test_semaphore_tracker() of test_multiprocessing_spawn fails randomly (race condition?)'
updated_at = <Date 2018-09-19.23:27:14.911>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2018-09-19.23:27:14.911>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-19.23:27:14.912>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-10-04.09:36:35.852>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31687
keywords = []
message_count = 9.0
messages = ['303678', '306557', '307355', '307356', '310360', '310949', '320613', '321016', '325825']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'vstinner', 'stevezh']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31687'
versions = ['Python 3.6', 'Python 3.7']
Activity
vstinner commentedon Oct 4, 2017
Recent example of the slow and busy (load of 8.05 with 2 CPUs) x86 Gentoo Refleaks 3.6 buildbot:
http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/111/steps/test/logs/stdio
vstinner commentedon Nov 20, 2017
Failure on x86 Gentoo Refleaks 3.6, two days ago:
http://buildbot.python.org/all/#/builders/72/builds/41
======================================================================
FAIL: test_semaphore_tracker (test.test_multiprocessing_spawn.TestSemaphoreTracker)
----------------------------------------------------------------------
======================================================================
FAIL: test_sys_exit (test.test_multiprocessing_spawn.WithProcessesTestSubclassingProcess)
----------------------------------------------------------------------
vstinner commentedon Dec 1, 2017
x86 Tiger 3.x:
http://buildbot.python.org/all/#/builders/30/builds/260
======================================================================
FAIL: test_semaphore_tracker (test.test_multiprocessing_forkserver.TestSemaphoreTracker)
----------------------------------------------------------------------
======================================================================
FAIL: test_forkserver_sigkill (test.test_multiprocessing_forkserver.WithProcessesTestProcess)
----------------------------------------------------------------------
subler commentedon Dec 1, 2017
It maybe the semaphore tracker process doesn't unlink the semaphore as soon as possible when child process exit when the system is slow. Increase test case's time in the child and before assertRaises will help.
vstinner commentedon Jan 20, 2018
To reproduce the issue, someone should increase the system load to make the system slower and make timeouts more likely, and change the order in which processes are run. I'm using the "stress" utility on Linux for that. But sometimes, the best stress tool... is the Python test suite, since running the full test suite in multiple processes in parallel (./python -m test -j0 -r) runs various workloads, and so sometimes trigger bugs which would be very hard to trigger othewise.
Terminal 1: run "./python -m test -j0 -r", stress or anything else to make the system slower
Terminal 2: run "./python -m test test_multiprocessing_fork -m test_semaphore_tracker -F -v" which runs the test in a loop until it fails.
ncoghlan commentedon Jan 28, 2018
A warning I'm sometimes seeing currently on successful test runs is as follows:
------
/home/ncoghlan/devel/cpython/Lib/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak.
warnings.warn('semaphore_tracker: process died unexpectedly, '
------
So it may be that the system has to be under load for the test suite to notice that failure.
vstinner commentedon Jun 27, 2018
See also bpo-33613.
vstinner commentedon Jul 4, 2018
This failure is unrelated to the semaphore tracker, so I created bpo-34040: "test_multiprocessing_forkserver: test_forkserver_sigkill() fails randomly".
vstinner commentedon Sep 19, 2018
bpo-33613 and bpo-34040 have been fixed. I hope that it will be enough.