And here is another candidate for the same reason:

[ 1053s] =================================== FAILURES ===================================
[ 1053s] _____________________________ test_worker_bad_args _____________________________
[ 1053s] 
[ 1053s] c = <Client: No scheduler connected>
[ 1053s] s = <Scheduler 'tcp://127.0.0.1:39999', workers: 0, cores: 0, tasks: 0>
[ 1053s] a = <Worker 'tcp://127.0.0.1:34669', name: 0, status: closed, stored: 0, running: 0/1, ready: 0, comm: 0, waiting: 0>
[ 1053s] b = <Worker 'tcp://127.0.0.1:34959', name: 1, status: closed, stored: 0, running: 0/2, ready: 0, comm: 0, waiting: 0>
[ 1053s] 
[ 1053s]     @gen_cluster(client=True)
[ 1053s]     async def test_worker_bad_args(c, s, a, b):
[ 1053s]         class NoReprObj:
[ 1053s]             """This object cannot be properly represented as a string."""
[ 1053s]     
[ 1053s]             def __str__(self):
[ 1053s]                 raise ValueError("I have no str representation.")
[ 1053s]     
[ 1053s]             def __repr__(self):
[ 1053s]                 raise ValueError("I have no repr representation.")
[ 1053s]     
[ 1053s]         x = c.submit(NoReprObj, workers=a.address)
[ 1053s]         await wait(x)
[ 1053s]         assert not a.executing_count
[ 1053s]         assert a.data
[ 1053s]     
[ 1053s]         def bad_func(*args, **kwargs):
[ 1053s]             1 / 0
[ 1053s]     
[ 1053s]         class MockLoggingHandler(logging.Handler):
[ 1053s]             """Mock logging handler to check for expected logs."""
[ 1053s]     
[ 1053s]             def __init__(self, *args, **kwargs):
[ 1053s]                 self.reset()
[ 1053s]                 super().__init__(*args, **kwargs)
[ 1053s]     
[ 1053s]             def emit(self, record):
[ 1053s]                 self.messages[record.levelname.lower()].append(record.getMessage())
[ 1053s]     
[ 1053s]             def reset(self):
[ 1053s]                 self.messages = {
[ 1053s]                     "debug": [],
[ 1053s]                     "info": [],
[ 1053s]                     "warning": [],
[ 1053s]                     "error": [],
[ 1053s]                     "critical": [],
[ 1053s]                 }
[ 1053s]     
[ 1053s]         hdlr = MockLoggingHandler()
[ 1053s]         old_level = logger.level
[ 1053s]         logger.setLevel(logging.DEBUG)
[ 1053s]         logger.addHandler(hdlr)
[ 1053s]         y = c.submit(bad_func, x, k=x, workers=b.address)
[ 1053s]         await wait(y)
[ 1053s]     
[ 1053s]         assert not b.executing_count
[ 1053s]         assert y.status == "error"
[ 1053s]         # Make sure job died because of bad func and not because of bad
[ 1053s]         # argument.
[ 1053s]         with pytest.raises(ZeroDivisionError):
[ 1053s]             await y
[ 1053s]     
[ 1053s]         tb = await y._traceback()
[ 1053s] >       assert any("1 / 0" in line for line in pluck(3, traceback.extract_tb(tb)) if line)
[ 1053s] E       assert False
[ 1053s] E        +  where False = any(<generator object test_worker_bad_args.<locals>.<genexpr> at 0x7f2604217840>)
[ 1053s] 
[ 1053s] distributed/tests/test_worker.py:143: AssertionError
[ 1053s] ----------------------------- Captured stdout call -----------------------------
[ 1053s] Dumped cluster state to test_cluster_dump/test_worker_bad_args.yaml
[ 1053s] ----------------------------- Captured stderr call -----------------------------
[ 1053s] 2022-03-25 22:34:16,577 - distributed.scheduler - INFO - Clear task state
[ 1053s] 2022-03-25 22:34:16,578 - distributed.scheduler - INFO -   Scheduler at:     tcp://127.0.0.1:39999
[ 1053s] 2022-03-25 22:34:16,579 - distributed.scheduler - INFO -   dashboard at:           127.0.0.1:42055
[ 1053s] 2022-03-25 22:34:16,583 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,585 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,586 - distributed.worker - INFO -          dashboard at:            127.0.0.1:40033
[ 1053s] 2022-03-25 22:34:16,587 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:39999
[ 1053s] 2022-03-25 22:34:16,588 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,589 - distributed.worker - INFO -               Threads:                          1
[ 1053s] 2022-03-25 22:34:16,590 - distributed.worker - INFO -                Memory:                  31.25 GiB
[ 1053s] 2022-03-25 22:34:16,591 - distributed.worker - INFO -       Local Directory: /home/abuild/rpmbuild/BUILD/distributed-2022.03.0/dask-worker-space/worker-0wbtc1nj
[ 1053s] 2022-03-25 22:34:16,592 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,593 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:34959
[ 1053s] 2022-03-25 22:34:16,595 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:34959
[ 1053s] 2022-03-25 22:34:16,596 - distributed.worker - INFO -          dashboard at:            127.0.0.1:43307
[ 1053s] 2022-03-25 22:34:16,597 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:39999
[ 1053s] 2022-03-25 22:34:16,598 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,599 - distributed.worker - INFO -               Threads:                          2
[ 1053s] 2022-03-25 22:34:16,600 - distributed.worker - INFO -                Memory:                  31.25 GiB
[ 1053s] 2022-03-25 22:34:16,601 - distributed.worker - INFO -       Local Directory: /home/abuild/rpmbuild/BUILD/distributed-2022.03.0/dask-worker-space/worker-nug7nicb
[ 1053s] 2022-03-25 22:34:16,602 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,654 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://127.0.0.1:34669', name: 0, status: undefined, memory: 0, processing: 0>
[ 1053s] 2022-03-25 22:34:16,679 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,679 - distributed.core - INFO - Starting established connection
[ 1053s] 2022-03-25 22:34:16,680 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://127.0.0.1:34959', name: 1, status: undefined, memory: 0, processing: 0>
[ 1053s] 2022-03-25 22:34:16,705 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34959
[ 1053s] 2022-03-25 22:34:16,706 - distributed.core - INFO - Starting established connection
[ 1053s] 2022-03-25 22:34:16,706 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:39999
[ 1053s] 2022-03-25 22:34:16,708 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,709 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:39999
[ 1053s] 2022-03-25 22:34:16,710 - distributed.worker - INFO - -------------------------------------------------
[ 1053s] 2022-03-25 22:34:16,711 - distributed.core - INFO - Starting established connection
[ 1053s] 2022-03-25 22:34:16,711 - distributed.core - INFO - Starting established connection
[ 1053s] 2022-03-25 22:34:16,750 - distributed.scheduler - INFO - Receive client connection: Client-b45025b6-ac8b-11ec-bbd8-18c04d911025
[ 1053s] 2022-03-25 22:34:16,775 - distributed.core - INFO - Starting established connection
[ 1053s] 2022-03-25 22:34:16,799 - distributed.worker - DEBUG - Ensure communicating. Pending: 1. Connections: 0/50
[ 1053s] 2022-03-25 22:34:16,801 - distributed.worker - DEBUG - Request 1 keys for task <TaskState 'bad_func-33f97bedb5ad8ac4769ef2a8e85d2c6f' waiting> from tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,805 - distributed.worker - WARNING - Compute Failed
[ 1053s] Key:       bad_func-33f97bedb5ad8ac4769ef2a8e85d2c6f
[ 1053s] Function:  bad_func
[ 1053s] args:      (< could not convert arg to str >)
[ 1053s] kwargs:    {'k': < could not convert arg to str >}
[ 1053s] Exception: "ZeroDivisionError('division by zero')"
[ 1053s] 
[ 1053s] 2022-03-25 22:34:16,806 - distributed.worker - DEBUG - Send compute response to scheduler: bad_func-33f97bedb5ad8ac4769ef2a8e85d2c6f, {'status': 'error', 'exception': <Serialize: b'\x80\x05\x95W\x03\x00\x00\x00\x00\x00\x00\x8c\x16tblib.pickling_support\x94\x8c\x12unpickle_exception\x94\x93\x94(\x8c\x08builtins\x94\x8c\x11ZeroDivisionError\x94\x93\x94\x8c\x10division by zero\x94\x85\x94Nh\x00\x8c\x12unpickle_traceback\x94\x93\x94\x8c\x05tblib\x94\x8c\x05Frame\x94\x93\x94)\x81\x94}\x94(\x8c\x08f_locals\x94}\x94\x8c\tf_globals\x94}\x94(\x8c\x08__name__\x94\x8c\x12distributed.worker\x94\x8c\x08__file__\x94\x8cG/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/worker.py\x94u\x8c\x06f_code\x94h\n\x8c\x04Code\x94\x93\x94)\x81\x94}\x94(\x8c\x0bco_filename\x94h\x16\x8c\x07co_name\x94\x8c\x15apply_function_simple\x94\x8c\x0bco_argcount\x94K\x00\x8c\x11co_kwonlyargcount\x94K\x00\x8c\x0bco_varnames\x94)\x8c\nco_nlocals\x94K\x00\x8c\x0cco_stacksize\x94K\x00\x8c\x08co_flags\x94K@\x8c\x0eco_firstlineno\x94K\x00ub\x8c\x08f_lineno\x94M\xf3\x10ubM\xf1\x10h\n\x8c\tTraceback\x94\x93\x94)\x81\x94}\x94(\x8c\x08tb_frame\x94h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(\x8c\x08__name__\x94\x8c\x0btest_worker\x94\x8c\x08__file__\x94\x8cR/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/tests/test_worker.py\x94uh\x17h\x19)\x81\x94}\x94(h\x1c\x8cR/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/tests/test_worker.py\x94h\x1d\x8c\x08bad_func\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&K\xcfub\x8c\ttb_lineno\x94K\xcfub\x87\x94R\x94t\x94R\x94.'>, 'traceback': <Serialize: b'\x80\x05\x95\xfe\x01\x00\x00\x00\x00\x00\x00\x8c\x16tblib.pickling_support\x94\x8c\x12unpickle_traceback\x94\x93\x94\x8c\x05tblib\x94\x8c\x05Frame\x94\x93\x94)\x81\x94}\x94(\x8c\x08f_locals\x94}\x94\x8c\tf_globals\x94}\x94(\x8c\x08__name__\x94\x8c\x0btest_worker\x94\x8c\x08__file__\x94\x8cR/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/tests/test_worker.py\x94u\x8c\x06f_code\x94h\x03\x8c\x04Code\x94\x93\x94)\x81\x94}\x94(\x8c\x0bco_filename\x94\x8cR/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/tests/test_worker.py\x94\x8c\x07co_name\x94\x8c\x08bad_func\x94\x8c\x0bco_argcount\x94K\x00\x8c\x11co_kwonlyargcount\x94K\x00\x8c\x0bco_varnames\x94)\x8c\nco_nlocals\x94K\x00\x8c\x0cco_stacksize\x94K\x00\x8c\x08co_flags\x94K@\x8c\x0eco_firstlineno\x94K\x00ub\x8c\x08f_lineno\x94K\xcfubK\xcfN\x87\x94R\x94.'>, 'exception_text': "ZeroDivisionError('division by zero')", 'traceback_text': '  File "/home/abuild/rpmbuild/BUILD/distributed-2022.03.0/distributed/tests/test_worker.py", line 207, in bad_func\n    return foo.x\n', 'op': 'task-erred', 'start': 1648247656.8289287, 'stop': 1648247656.8295643, 'thread': 139801311307328, 'key': 'bad_func-33f97bedb5ad8ac4769ef2a8e85d2c6f'}
[ 1053s] 2022-03-25 22:34:16,899 - distributed.scheduler - INFO - Remove client Client-b45025b6-ac8b-11ec-bbd8-18c04d911025
[ 1053s] 2022-03-25 22:34:16,900 - distributed.scheduler - INFO - Remove client Client-b45025b6-ac8b-11ec-bbd8-18c04d911025
[ 1053s] 2022-03-25 22:34:16,901 - distributed.scheduler - INFO - Close client connection: Client-b45025b6-ac8b-11ec-bbd8-18c04d911025
[ 1053s] 2022-03-25 22:34:16,902 - distributed.worker - DEBUG - Release key {'key': 'NoReprObj-094fb790cb13239ef4983171124ba30b', 'cause': None, 'stimulus_id': 'memory-released-1648247656.9005365'}
[ 1053s] 2022-03-25 22:34:16,904 - distributed.worker - DEBUG - Release key {'key': 'NoReprObj-094fb790cb13239ef4983171124ba30b', 'cause': None, 'stimulus_id': 'memory-released-1648247656.9005365'}
[ 1053s] 2022-03-25 22:34:16,905 - distributed.worker - DEBUG - Release key {'key': 'bad_func-33f97bedb5ad8ac4769ef2a8e85d2c6f', 'cause': None, 'stimulus_id': 'erred-released-1648247656.9005625'}
[ 1053s] 2022-03-25 22:34:16,908 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,909 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34959
[ 1053s] 2022-03-25 22:34:16,911 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://127.0.0.1:34669', name: 0, status: closing, memory: 0, processing: 0>
[ 1053s] 2022-03-25 22:34:16,912 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:34669
[ 1053s] 2022-03-25 22:34:16,912 - distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://127.0.0.1:34959', name: 1, status: closing, memory: 0, processing: 0>
[ 1053s] 2022-03-25 22:34:16,913 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:34959
[ 1053s] 2022-03-25 22:34:16,913 - distributed.scheduler - INFO - Lost all workers
[ 1053s] 2022-03-25 22:34:16,914 - distributed.scheduler - INFO - Scheduler closing...
[ 1053s] 2022-03-25 22:34:16,915 - distributed.scheduler - INFO - Scheduler closing all comms
[ 1053s] =============================== warnings summary ===============================

Both of them have been mentioned by @mcepl in #5709 (comment) although that might have been buried since it wasn't the right place.

Uh oh!

Python 3.10 support #5460

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions