tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7ff0de6dfa90>>, <Task finished name='Task-96079' coro=<Worker.gather_dep() done, defined at /home/travis/build/dask/distributed/distributed/worker.py:1960> exception=CommClosedError()>)
2278Traceback (most recent call last):
2279  File "/home/travis/build/dask/distributed/distributed/worker.py", line 1980, in gather_dep
2280    self.validate_state()
2281  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2912, in validate_state
2282    self.validate_task(ts)
2283  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2867, in validate_task
2284    self.validate_task_flight(ts)
2285  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2854, in validate_task_flight
2286    assert ts.key in self.in_flight_workers[ts.coming_from]
2287AssertionError
2288
2289During handling of the above exception, another exception occurred:
2290
2291Traceback (most recent call last):
2292  File "/home/travis/miniconda/envs/dask-distributed/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
2293    ret = callback()
2294  File "/home/travis/miniconda/envs/dask-distributed/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
2295    future.result()
2296  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2082, in gather_dep
2297    self.batched_stream.send(
2298  File "/home/travis/build/dask/distributed/distributed/batched.py", line 146, in send
2299    raise CommClosedError
2300distributed.comm.core.CommClosedError
2301distributed.worker - ERROR - Handle missing dep failed, retrying
2302Traceback (most recent call last):
2303  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2124, in handle_missing_dep
2304    self.bad_dep(dep)
2305  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2105, in bad_dep
2306    "Could not find dependent %s.  Check worker logs" % str(dep.ts)
2307AttributeError: 'TaskState' object has no attribute 'ts'
2308distributed.worker - ERROR - Handle missing dep failed, retrying
2309Traceback (most recent call last):
2310  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2124, in handle_missing_dep
2311    self.bad_dep(dep)
2312  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2105, in bad_dep
2313    "Could not find dependent %s.  Check worker logs" % str(dep.ts)
2314AttributeError: 'TaskState' object has no attribute 'ts'
2315
2316

distributed.worker - ERROR - Handle missing dep failed, retrying
2309Traceback (most recent call last):
2310  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2124, in handle_missing_dep
2311    self.bad_dep(dep)
2312  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2105, in bad_dep
2313    "Could not find dependent %s.  Check worker logs" % str(dep.ts)
2314AttributeError: 'TaskState' object has no attribute 'ts'
2315
2316During handling of the above exception, another exception occurred:
2317
2318Traceback (most recent call last):
2319  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in handle_missing_dep
2320    deps = {dep for dep in deps if dep.dependents}
2321  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in <setcomp>
2322    deps = {dep for dep in deps if dep.dependents}
2323AttributeError: 'Worker' object has no attribute 'dependents'
2324distributed.worker - ERROR - Handle missing dep failed, retrying
2325Traceback (most recent call last):
2326  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2124, in handle_missing_dep
2327    self.bad_dep(dep)
2328  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2105, in bad_dep
2329    "Could not find dependent %s.  Check worker logs" % str(dep.ts)
2330AttributeError: 'TaskState' object has no attribute 'ts'
2331
2332During handling of the above exception, another exception occurred:
2333
2334Traceback (most recent call last):
2335  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in handle_missing_dep
2336    deps = {dep for dep in deps if dep.dependents}
2337  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in <setcomp>
2338    deps = {dep for dep in deps if dep.dependents}
2339AttributeError: 'Worker' object has no attribute 'dependents'
2340
2341During handling of the above exception, another exception occurred:
2342
2343Traceback (most recent call last):
2344  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in handle_missing_dep
2345    deps = {dep for dep in deps if dep.dependents}
2346  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in <setcomp>
2347    deps = {dep for dep in deps if dep.dependents}
2348AttributeError: 'Worker' object has no attribute 'dependents'
2349distributed.worker - ERROR - Handle missing dep failed, retrying
2350Traceback (most recent call last):
2351  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2124, in handle_missing_dep
2352    self.bad_dep(dep)
2353  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2105, in bad_dep
2354    "Could not find dependent %s.  Check worker logs" % str(dep.ts)
2355AttributeError: 'TaskState' object has no attribute 'ts'
2356
2357During handling of the above exception, another exception occurred:
2358
2359Traceback (most recent call last):
2360  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in handle_missing_dep
2361    deps = {dep for dep in deps if dep.dependents}
2362  File "/home/travis/build/dask/distributed/distributed/worker.py", line 2117, in <setcomp>
2363    deps = {dep for dep in deps if dep.dependents}
2364AttributeError: 'Worker' object has no attribute 'dependents'
2365
2366During handling of the above exception, another exception occurred: