Process never exits due to hang in shutdown #2284
Description
We have a process that sometimes fails to exit until we hit ctrl-c. Traceback at that time shows the process is hung, attempting to "join" a worker thread:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/opentelemetry/sdk/trace/__init__.py", line 1136, in shutdown
self._active_span_processor.shutdown()
File "/usr/local/lib/python3.7/dist-packages/opentelemetry/sdk/trace/__init__.py", line 166, in shutdown
sp.shutdown()
File "/usr/local/lib/python3.7/dist-packages/opentelemetry/sdk/trace/export/__init__.py", line 385, in shutdown
self.worker_thread.join()
File "/usr/lib/python3.7/threading.py", line 1044, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.7/threading.py", line 1060, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
In this case, the process completed in about 500 ms and would have produced only a few spans. The process was configured to export to a remote OT collector which may or may not have been reachable at the time - but we expect the "reachability" of the collector to not interfere with process execution. We gave it like 30 seconds to complete before terminating the process.
This, of course, interferes with auto-scaling and our ability to manage processes.
Please advise.