py<=3.10: Green logging locks by tipabu · Pull Request #754 · eventlet/eventlet

tipabu · 2022-04-20T19:52:07Z

gc.get_referrers() returns nothing but lists and dicts, so previously we'd always trip a TypeError and skip patching.

Closes #730

This also seems to allow _fix_py3_lock to actually have an effect on py34-py39, though finding the locks that need fixing remains a challenge.

codecov · 2022-04-20T19:54:46Z

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (33e05ca) 53% compared to head (bf2fe30) 53%.

Files	Patch %	Lines
eventlet/patcher.py	84%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##           master   #754   +/-   ##
=====================================
  Coverage      53%    53%           
=====================================
  Files          88     88           
  Lines        9862   9874   +12     
  Branches     1845   1849    +4     
=====================================
+ Hits         5320   5331   +11     
+ Misses       4157   4156    -1     
- Partials      385    387    +2

Flag	Coverage Δ
ipv6	`22% <0%> (-1%)`	⬇️
py310epolls	`52% <0%> (-1%)`	⬇️
py310poll	`52% <0%> (-1%)`	⬇️
py310selects	`52% <0%> (-1%)`	⬇️
py311epolls	`52% <7%> (-1%)`	⬇️
py312epolls	`50% <7%> (-1%)`	⬇️
py38epolls	`52% <69%> (+<1%)`	⬆️
py38openssl	`51% <69%> (+<1%)`	⬆️
py38poll	`52% <69%> (+<1%)`	⬆️
py38selects	`52% <69%> (+<1%)`	⬆️
py39dnspython1	`50% <69%> (+<1%)`	⬆️
py39epolls	`52% <69%> (+<1%)`	⬆️
py39poll	`52% <69%> (+<1%)`	⬆️
py39selects	`52% <69%> (+<1%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tipabu · 2022-04-21T05:21:13Z

eventlet/patcher.py

+        # https://github.com/eventlet/eventlet/issues/546 -- so green a handful
+        # that we know are significant
+        import logging
+        _fix_py3_rlock(logging._lock)


Maybe should add in something like

logging._acquireLock() try: for ref in logging._handlerList: handler = ref() if handler: _fix_py3_rlock(handler.lock) del handler finally: logging._releaseLock()

?

gc.get_referrers() returns nothing but lists and dicts, so previously we'd always trip a TypeError and skip patching. Closes eventlet#730

See also: eventlet#546

temoto · 2022-05-04T11:16:13Z

@tipabu thanks a lot for fix. I'm going to pull remote verification card here. Did you run anything close to real project with this patch?

tipabu · 2022-05-10T17:24:36Z

Did you run anything close to real project with this patch?

Not yet, unfortunately. My prod environment is still in the process of transitioning from py27 -> py37, so any significant testing with py310 is still off in the nebulous future. I could probably get it running in my homelab, but haven't had time and I'm not entirely sure how I'd want to validate it.

For a little more context, we're trying to remove some monkey_patch()-as-import-side-effect so we can be more intentioned about when we're monkey-patched and when we're not. As part of that, we got a little skittish about the impact of delaying monkey-patching -- #546 specifically was identified as a concern, in part because we currently explicitly green logging._lock, and it's unclear what impact there may be in taking that out.

thomasgoirand · 2022-09-12T12:14:44Z

Hi,

FYI, this fixed the unit test failure for me with Python 3.10 in Debian unstable. Thanks for the patch!

Cheers,

Thomas Goirand (zigo)

thomasgoirand · 2022-09-14T15:36:05Z

Hi,

Unfortunately, I had to revert this patch in Eventlet. Indeed, it's breaking a number of OpenStack packages during discovery. An example with watcher:

Failures during discovery

--- import errors ---
Failed to import test module: watcher.tests.cmd.test_api
Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/<<PKGBUILDDIR>>/watcher/tests/cmd/test_api.py", line 25, in <module>
    from watcher.cmd import api
  File "/<<PKGBUILDDIR>>/watcher/cmd/__init__.py", line 20, in <module>
    eventlet.monkey_patch()
  File "/usr/lib/python3/dist-packages/eventlet/patcher.py", line 280, in monkey_patch
    _green_existing_locks()
  File "/usr/lib/python3/dist-packages/eventlet/patcher.py", line 413, in _green_existing_locks
    _fix_py3_rlock(obj)
  File "/usr/lib/python3/dist-packages/eventlet/patcher.py", line 443, in _fix_py3_rlock
    if v == old:
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 277, in __eq__
    return self.__dict__ == other.__dict__
AttributeError: '_thread.RLock' object has no attribute '__dict__'

up: format

tipabu · 2022-11-04T18:51:40Z

Looks like a testtools bug -- proposed testing-cabal/testtools#336 to address. In the meantime, this patch should be fine to use is instead of ==.

temoto · 2022-11-17T17:14:21Z

@thomasgoirand could you confirm that new version using is comparison works?

https://build.opensuse.org/request/show/1059026 by user dgarcia + dimstar_suse - Add fix-py3-rlock.patch to make the code compatible with python 3.11, gh#eventlet/eventlet#754

4383 · 2023-12-15T09:10:15Z

@thomasgoirand: o/ please can you confirm that new version using is comparison works?

See previous comments

4383

LGTM

4383 · 2023-12-19T11:21:11Z

@itamarst: Feel free to merge this patch if you think that it is now ok, else do not hesitate to request an other review.

Thanks

Several important and urgent fixes are released there. 0.34.3 ====== eventlet/eventlet#875 * Fix security issue in the wsgi module related to RFC 9112 eventlet/eventlet#826 * Fix segfault, a new approach for greening existing locks eventlet/eventlet#866 * greendns: fix getaddrinfo parameter name eventlet/eventlet#809 * Fix deprecation warning on ssl.PROTOCOL_TLS eventlet/eventlet#872 * Pytests, fix error at teardown of TestGreenSocket.test_full_duplex eventlet/eventlet#871 * Skip test which uses Py cgi module eventlet/eventlet#865 * Drop old code based on python < 3.7.34.2 ====== eventlet/eventlet#861 * Allowing inheritance of GreenSSLSocket without overriding the __new_ method eventlet/eventlet#796 * [bug] Fix broken API related to `__version__` removal eventlet/eventlet#859 * [doc] Fix pypi broken link eventlet/eventlet#857 0.34.1 ====== eventlet/eventlet#842 * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: Ib2e59a207b86ae90fa391bf1dff7819851dc9c9b

* Update requirements from branch 'master' to 827a86739e66ffb537b5ea7a65a3cc74eb3fabf1 - Merge "Update eventlet to 0.34.3" - Update eventlet to 0.34.3 Several important and urgent fixes are released there. 0.34.3 ====== eventlet/eventlet#875 * Fix security issue in the wsgi module related to RFC 9112 eventlet/eventlet#826 * Fix segfault, a new approach for greening existing locks eventlet/eventlet#866 * greendns: fix getaddrinfo parameter name eventlet/eventlet#809 * Fix deprecation warning on ssl.PROTOCOL_TLS eventlet/eventlet#872 * Pytests, fix error at teardown of TestGreenSocket.test_full_duplex eventlet/eventlet#871 * Skip test which uses Py cgi module eventlet/eventlet#865 * Drop old code based on python < 3.7.34.2 ====== eventlet/eventlet#861 * Allowing inheritance of GreenSSLSocket without overriding the __new_ method eventlet/eventlet#796 * [bug] Fix broken API related to `__version__` removal eventlet/eventlet#859 * [doc] Fix pypi broken link eventlet/eventlet#857 0.34.1 ====== eventlet/eventlet#842 * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: Ib2e59a207b86ae90fa391bf1dff7819851dc9c9b

* Update requirements from branch 'master' to 08f829d8375b4059af365191e0907069be9fb739 - Update eventlet to 0.35.0 0.35.0 ====== eventlet/eventlet#897 * [fix] fix truncate size nullable eventlet/eventlet#789 * [fix] Handle transport endpoint shutdown in conditions eventlet/eventlet#884 * [fix] Rework reject_bad_requests option eventlet/eventlet#890 * [fix] Fix NameError introduced by #826 eventlet/eventlet#890 * [feature] Support awaiting GreenThread in an `async def` context eventlet/eventlet#889 * [feature] Asyncio hub support for Python 3.7 to 3.9 eventlet/eventlet#886 * [fix] Fix bad exceptions handlings eventlet/eventlet#883 * [feature] Support using asyncio coroutines from inside greenlets eventlet/eventlet#877 * [removal] Remove deprecated CGIHTTPServer and SimpleHTTPServer eventlet/eventlet#881 * [feature] Add an asyncio hub for eventlet eventlet/eventlet#870 0.34.3 ====== eventlet/eventlet#875 * Fix security issue in the wsgi module related to RFC 9112 eventlet/eventlet#826 * Fix segfault, a new approach for greening existing locks eventlet/eventlet#866 * greendns: fix getaddrinfo parameter name eventlet/eventlet#809 * Fix deprecation warning on ssl.PROTOCOL_TLS eventlet/eventlet#872 * Pytests, fix error at teardown of TestGreenSocket.test_full_duplex eventlet/eventlet#871 * Skip test which uses Py cgi module eventlet/eventlet#865 * Drop old code based on python < 3.7.34.2 ====== eventlet/eventlet#861 * Allowing inheritance of GreenSSLSocket without overriding the __new_ method eventlet/eventlet#796 * [bug] Fix broken API related to `__version__` removal eventlet/eventlet#859 * [doc] Fix pypi broken link eventlet/eventlet#857 0.34.1 ====== eventlet/eventlet#842 * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: I909be1d1812eaed574525866dbc123083684571d

0.35.0 ====== eventlet/eventlet#897 * [fix] fix truncate size nullable eventlet/eventlet#789 * [fix] Handle transport endpoint shutdown in conditions eventlet/eventlet#884 * [fix] Rework reject_bad_requests option eventlet/eventlet#890 * [fix] Fix NameError introduced by #826 eventlet/eventlet#890 * [feature] Support awaiting GreenThread in an `async def` context eventlet/eventlet#889 * [feature] Asyncio hub support for Python 3.7 to 3.9 eventlet/eventlet#886 * [fix] Fix bad exceptions handlings eventlet/eventlet#883 * [feature] Support using asyncio coroutines from inside greenlets eventlet/eventlet#877 * [removal] Remove deprecated CGIHTTPServer and SimpleHTTPServer eventlet/eventlet#881 * [feature] Add an asyncio hub for eventlet eventlet/eventlet#870 0.34.3 ====== eventlet/eventlet#875 * Fix security issue in the wsgi module related to RFC 9112 eventlet/eventlet#826 * Fix segfault, a new approach for greening existing locks eventlet/eventlet#866 * greendns: fix getaddrinfo parameter name eventlet/eventlet#809 * Fix deprecation warning on ssl.PROTOCOL_TLS eventlet/eventlet#872 * Pytests, fix error at teardown of TestGreenSocket.test_full_duplex eventlet/eventlet#871 * Skip test which uses Py cgi module eventlet/eventlet#865 * Drop old code based on python < 3.7.34.2 ====== eventlet/eventlet#861 * Allowing inheritance of GreenSSLSocket without overriding the __new_ method eventlet/eventlet#796 * [bug] Fix broken API related to `__version__` removal eventlet/eventlet#859 * [doc] Fix pypi broken link eventlet/eventlet#857 0.34.1 ====== eventlet/eventlet#842 * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: I909be1d1812eaed574525866dbc123083684571d

Several important and urgent fixes are released there. 0.34.1 ====== * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: Ib2e59a207b86ae90fa391bf1dff7819851dc9c9b

Several important and urgent fixes are released there. 0.34.2 ====== eventlet/eventlet#861 * Allowing inheritance of GreenSSLSocket without overriding the __new_ method eventlet/eventlet#796 * [bug] Fix broken API related to `__version__` removal eventlet/eventlet#859 * [doc] Fix pypi broken link eventlet/eventlet#857 0.34.1 ====== eventlet/eventlet#842 * [bug] Fix memory leak in greendns eventlet/eventlet#810 * [infra] Fix OIDC authentication failure eventlet/eventlet#855 * [bug] Ignore asyncore and asynchat for Python 3.12+ eventlet/eventlet#804 0.34.0 (Not released on Pypi but landed with 0.34.1) ==================================================== * Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' eventlet/eventlet#847 * Add support of Python 3.12 eventlet/eventlet#817 * Drop unmaintained and unused stdlib tests eventlet/eventlet#820 * Fix tests and CI for Python 3.7 and higher eventlet/eventlet#831 and eventlet/eventlet#832 * Stop claiming to create universal wheels eventlet/eventlet#841 * Fix green logging locks for Python versions <= 3.10 eventlet/eventlet#754 Change-Id: Ib2e59a207b86ae90fa391bf1dff7819851dc9c9b

tipabu commented Apr 21, 2022

View reviewed changes

tipabu added 2 commits May 2, 2022 13:07

py310: Fix _fix_py3_rlock

cd25321

gc.get_referrers() returns nothing but lists and dicts, so previously we'd always trip a TypeError and skip patching. Closes eventlet#730

py3: Green logging locks upon monkey-patching

e7c6887

See also: eventlet#546

tipabu force-pushed the py3-rlock branch from ec2b808 to e7c6887 Compare May 2, 2022 20:07

temoto added 2 commits May 4, 2022 13:05

test: guard isolated test run on collection

f8ed6e1

pep8

d138020

Use is instead of ==

0a5f6fa

Merge branch 'master' into py3-rlock

f23e805

temoto mentioned this pull request Jan 22, 2023

Python3.10 support #780

Closed

4383 mentioned this pull request Dec 12, 2023

Try to make CI pass tests #823

Closed

Merge branch 'master' into py3-rlock

65f4ed2

itamarst added important bug labels Dec 18, 2023

Pass correct argument, use identity comparison

5e48998

itamarst mentioned this pull request Dec 18, 2023

New eventlet release 0.34.1 #842

Closed

tipabu changed the title ~~py310: Fix _fix_py3_rlock~~ py<=3.10: Green logging locks Dec 18, 2023

pythonspeed added 2 commits December 18, 2023 16:15

More direct test

3261769

3.10 works fine, and we're not supporting Python 2 anymore

bf2fe30

itamarst requested a review from 4383 December 18, 2023 21:19

4383 approved these changes Dec 19, 2023

View reviewed changes

itamarst merged commit 72a29b6 into eventlet:master Dec 19, 2023

tipabu mentioned this pull request Dec 29, 2023

Greening existing locks doesn't always work right #864

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

py<=3.10: Green logging locks#754

py<=3.10: Green logging locks#754
itamarst merged 10 commits intoeventlet:masterfrom
tipabu:py3-rlock

tipabu commented Apr 20, 2022

Uh oh!

codecov bot commented Apr 20, 2022 •

edited

Loading

Uh oh!

tipabu Apr 21, 2022

Uh oh!

temoto commented May 4, 2022

Uh oh!

tipabu commented May 10, 2022

Uh oh!

thomasgoirand commented Sep 12, 2022

Uh oh!

thomasgoirand commented Sep 14, 2022 •

edited by temoto

Loading

Uh oh!

tipabu commented Nov 4, 2022

Uh oh!

temoto commented Nov 17, 2022

Uh oh!

4383 commented Dec 15, 2023

Uh oh!

4383 left a comment

Uh oh!

4383 commented Dec 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

tipabu commented Apr 20, 2022

Uh oh!

codecov bot commented Apr 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tipabu Apr 21, 2022

Choose a reason for hiding this comment

Uh oh!

temoto commented May 4, 2022

Uh oh!

tipabu commented May 10, 2022

Uh oh!

thomasgoirand commented Sep 12, 2022

Uh oh!

thomasgoirand commented Sep 14, 2022 • edited by temoto Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tipabu commented Nov 4, 2022

Uh oh!

temoto commented Nov 17, 2022

Uh oh!

4383 commented Dec 15, 2023

Uh oh!

4383 left a comment

Choose a reason for hiding this comment

Uh oh!

4383 commented Dec 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Apr 20, 2022 •

edited

Loading

thomasgoirand commented Sep 14, 2022 •

edited by temoto

Loading