Installation of dnspython 2.0 breaks seemingly unrelated things #558
Description
Hi there,
I apologize in advance for the weird bug -- but maybe you have an idea that can help us.
We run a Django-based web app, which we run in gunicorn using eventlet workers. Our app makes use of dnspython. After the upgrade to 2.0.0, our production server does no longer start because of this issue:
[2020-07-29 15:41:10 +0000] [4635] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
super(EventletWorker, self).init_process()
File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.8/site-packages/gunicorn/util.py", line 350, in import_app
__import__(module)
File "/src/pretix/src/pretix/wsgi.py", line 18, in <module>
application = Cling(MediaCling(get_wsgi_application()))
File "/usr/local/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 76, in __getattr__
self._setup(name)
File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 142, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.8/site-packages/pretixeu/production_settings.py", line 4, in <module>
from pretix.settings import *
File "/src/pretix/src/pretix/settings.py", line 8, in <module>
from kombu import Queue
File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
File "/usr/local/lib/python3.8/site-packages/kombu/__init__.py", line 81, in __getattr__
module = __import__(object_origins[name], None, None, [name])
File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 9, in <module>
from .serialization import prepare_accept_content
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 456, in <module>
for ep, args in entrypoints('kombu.serializers'): # pragma: no cover
File "/usr/local/lib/python3.8/site-packages/kombu/utils/compat.py", line 93, in entrypoints
for ep in importlib_metadata.entry_points().get(namespace, [])
File "/usr/local/lib/python3.8/importlib/metadata.py", line 542, in entry_points
ordered = sorted(eps, key=by_group)
File "/usr/local/lib/python3.8/importlib/metadata.py", line 540, in <genexpr>
dist.entry_points for dist in distributions())
File "/usr/local/lib/python3.8/importlib/metadata.py", line 240, in entry_points
return EntryPoint._from_text(self.read_text('entry_points.txt'))
File "/usr/local/lib/python3.8/importlib/metadata.py", line 491, in read_text
return self._path.joinpath(filename).read_text(encoding='utf-8')
File "/usr/local/lib/python3.8/pathlib.py", line 1232, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
File "/usr/local/lib/python3.8/pathlib.py", line 1218, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/usr/local/lib/python3.8/pathlib.py", line 1074, in _opener
return self._accessor.open(self, flags, mode)
File "/usr/local/lib/python3.8/site-packages/eventlet/green/os.py", line 107, in open
fd = __original_open__(file, flags, mode, dir_fd=dir_fd)
TypeError: open: path should be string, bytes or os.PathLike, not _NormalAccessor
[2020-07-29 15:41:10 +0000] [4635] [INFO] Worker exiting (pid: 4635)
Apparently eventlet does weird things to os. As you can see, dnspython does not appear in the traceback at all. Involved are gunicorn, django, kombu, importlib, eventlib, and our own code (pretix and pretixeu).
Yet, regardless of how many times I try, it reproducibly only happens if dnspython 2.0 is installed. I literally SSH'd into our production system, jumped into the docker container and execute things like this:
$ pip install dnspython==1.16.0
$ gunicorn …
works
$ pip install dnspython==2.0.0
Collecting dnspython==2.0.0
Using cached dnspython-2.0.0-py3-none-any.whl (208 kB)
Installing collected packages: dnspython
Attempting uninstall: dnspython
Found existing installation: dnspython 1.16.0
Uninstalling dnspython-1.16.0:
Successfully uninstalled dnspython-1.16.0
Successfully installed dnspython-2.0.0
$ gunicorn …
does not work
$ pip install dnspython==1.16.0
Collecting dnspython==1.16.0
Using cached dnspython-1.16.0-py2.py3-none-any.whl (188 kB)
Installing collected packages: dnspython
Attempting uninstall: dnspython
Found existing installation: dnspython 2.0.0
Uninstalling dnspython-2.0.0:
Successfully uninstalled dnspython-2.0.0
Successfully installed dnspython-1.16.0
$ gunicorn …
works
Our gunicorn line looks like this:
gunicorn pretix.wsgi \
--name pretix \
--workers $NUM_WORKERS \
--worker-class eventlet \
--max-requests 1200 \
--max-requests-jitter 50 \
--log-level=info \
--timeout 600 \
--bind=unix:/tmp/pretix.sock
I understand this is likely not dnspython's fault – but do you have any idea what could have caused it?