Repository navigation
Potential deadlock in ReaderWriterLock #86
Description
This behavior has been observed in the OpenStack TaskFlow project (https://opendev.org/openstack/taskflow)
After updating from fasteners 0.14.1 to 0.16.3, we have observed some issues with the TaskFlow CI, some tests are blocked with a deadlock.
I wrote a simple reproducer in the fasteners test: gthiemonge@6d40296
Basically, a first thread requests a writer lock, then a reentrant reader lock, while a second thread requests only the writer lock.
The test deadlocks after a few iterations:
$ pytest -k test_deadlock_reproducer -v -s tests/test_lock.py
============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.10.1, pytest-6.2.4, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/user/sources/fasteners
collected 18 items / 17 deselected / 1 selected
tests/test_lock.py::test_deadlock_reproducer thread1 iteration: 0
thread1 waiting for write lock
thread1 got write lock
thread1 waiting for read lock
thread1 got read lock
thread1 released read lock
thread1 released write lock
thread1 iteration: 1
thread1 waiting for write lock
thread1 got write lock
thread1 waiting for read lock
thread1 got read lock
thread1 released read lock
thread1 released write lock
thread1 iteration: 2
thread1 waiting for write lock
thread1 got write lock
thread1 waiting for read lock
thread2 iteration: 0
thread2 waiting for write lock
thread1 got read lock
thread1 released read lock
thread1 released write lock
thread2 got write lock
thread1 iteration: 3
thread2 released write lock
thread2 iteration: 1
thread1 waiting for write lock
thread2 waiting for write lock
thread1 got write lock
thread1 waiting for read lock
^C
The following commit 75cf59a triggers this behavior, reverting the commit seems to fix the issue.
Any ideas on how to fix it?
Activity
psarka commentedon Jan 25, 2022
Thanks for a great bug report! I'm unfamiliar with this part of the codebase, so it will take some time. @obondarev, maybe you would like to take a look?
obondarev commentedon Jan 25, 2022
Sure, I think read lock should be allowed if current thread already has a write lock. This means line #184 in [1] should be
"elif (self._writer == me) or not self.has_pending_writers:"
Can you please try that @gthiemonge ?
[1] 75cf59a
psarka commentedon Jan 25, 2022
I gave it a go, all tests pass, including the reproducer by @gthiemonge. Thanks 👍
I will take another carefull look at that part of the code and will release the fix soonish.
obondarev commentedon Jan 25, 2022
Thanks, @psarka !
gthiemonge commentedon Jan 25, 2022
Yes, it fixes my reproducer and our tests in TaskFlow.
Thanks Folks!
psarka commentedon Jan 25, 2022
Made a new release (0.17.3) with the fix.
jakirkham commentedon Jan 28, 2022
We are still seeing deadlocks with 0.17.3. Will try to come up with a reproducer.