PersistentDataset tempfile issue #2146
Description
Describe the bug
issue detected in v0.5.2:
Line 253 in feb3a33
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/monai/monai/data/dataset.py", line 93, in __getitem__
return self._transform(index)
File "/opt/monai/monai/data/dataset.py", line 259, in _transform
pre_random_item = self._cachecheck(self.data[index])
File "/opt/monai/monai/data/dataset.py", line 253, in _cachecheck
temp_hash_file.rename(hashfile)
File "/opt/conda/lib/python3.8/pathlib.py", line 1358, in rename
self._accessor.rename(self, target)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpc9cd7k66/0e709abea045c1af2a5a94d166f81d65.pt' -> 'val_cache/0e709abea045c1af2a5a94d166f81d65.pt'
root cause is that the default tempfile location /tmp is on a different filesystem than val_cache.
a potential workaround would be changing the default temp location of python:
export TMPDIR=/path/to/val_cache