Could not acquire workspace lock #2113
Description
Hi,
I was following the instruction to start a distributed network with dask-scheduler and dask-worker on a SLURM based system:
dask-scheduler --scheduler-file=$CSCRATCH/sche.json
srun -n 32 python `which dask-worker` --scheduler-file=$CSCRATCH/sche.json
The scheuler launched, but when I launched the workers I see many errors like this:
distributed.diskutils - ERROR - Could not acquire workspace lock on path: /global/u1/y/yfeng1/dask-worker-space/worker-yhmlzrrc.dirlock .Continuing without lock. This may result in workspaces not being cleaned up
Traceback (most recent call last):
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/diskutils.py", line 57, in __init__
with workspace._global_lock():
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/locket.py", line 194, in __enter__
self.acquire()
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/locket.py", line 188, in acquire
self._lock.acquire(self._timeout, self._retry_period)
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/locket.py", line 118, in acquire
lock.acquire(timeout, retry_period)
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/locket.py", line 162, in acquire
_lock_file_blocking(self._file)
File "/global/homes/y/yfeng1/.conda/envs/bccp/lib/python3.6/site-packages/distributed/locket.py", line 52, in _lock_file_blocking
fcntl.flock(file_.fileno(), fcntl.LOCK_EX)
OSError: [Errno 524] Unknown error 524
This error doesn't appear to be critical, as the workers proceeded to launch. But what is a workspace (documentation of distributed didn't mention it), how much data is exchanged in this directory, and how can I redirect the direct to a more scalable location?