Did some research:

We can have large temporary data here
SIGKILL will make it linger
The workers will try to get a lock for their directory

dask/distributed#1543
https://stackoverflow.com/questions/48657374/what-is-the-default-directory-where-dask-workers-store-results-or-files
https://distributed.readthedocs.io/en/latest/worker.html#spill-data-to-disk
dask/distributed#1540
dask/distributed#2113

Is this a different directory?

dask/dask#1659

Deactivate dumping data there. I guess we should do that because we read from files anyway and it should never be necessary, if I understand correctly. I'd rather have workers die than clutter disk space.

http://dask.pydata.org/en/latest/setup/hpc.html#no-local-storage

So that would mean:

Should it persist between sessions? No.
Might it contain temporary data of significant size that we should delete in case LiberTEM doesn't exit cleanly? YES -- but not for LiberTEM PROVIDED everything works as it should!
Does it contain things like lock files that are relevant for concurrent sessions or that might mess things up if they linger in case LiberTEM didn't exit cleanly? No, since workers only lock their own dir. But the file system needs to support locking.
Can two instances share the same folder? Yes, it seems, since each worker creates their own unique folder.
Might data accumulate over time, like log files? Yes, leftover dirs accumulate when the worker (or scheduler?) dies unexpectedly.
Is the IO performance or latency of the underlying file system performance-critical somehow? YES, if it is used for large intermediate data. Should not be relevant for LiberTEM IF everything works as expected.

What about using %TEMP%\libertem\dask on Windows? That expands to C:\Users\<user>\AppData\Local\Temp\libertem\dask on my system (Windows 10).

Don't litter cwd with dask-worker-space #80

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions