-
Notifications
You must be signed in to change notification settings - Fork 1.5k
SmartCache is not working in Clara Train #1852
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
SmartCache is not working when used in an MMAR of Clara Train despite using SmartCacheHandler as the first handler.
To Reproduce
Steps to reproduce the behavior:
- Take a Clara Train MMAR
- Use
SmartCachePatchWSIDatasetas the dataset. - Run MMAR
Additional context
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/workspace/clara-train/pt/src/medl/apps/train.py", line 35, in <module>
main()
File "/workspace/clara-train/pt/src/medl/apps/train.py", line 27, in main
train_mmar(args)
File "/workspace/clara-train/pt/src/medl/apps/mmar_conf.py", line 28, in train_mmar
trainer.run()
File "/workspace/monai/monai/engines/trainer.py", line 48, in run
super().run()
File "/workspace/monai/monai/engines/workflow.py", line 197, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
return self._internal_run()
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
self._handle_exception(e)
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/workspace/monai/monai/handlers/checkpoint_saver.py", line 274, in exception_raised
raise e
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
time_taken = self._run_once_on_dataset()
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
self._handle_exception(e)
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/workspace/monai/monai/handlers/checkpoint_saver.py", line 274, in exception_raised
raise e
File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
self.state.batch = next(self._dataloader_iter)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 519, in __next__
data = self._next_data()
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1169, in _next_data
return self._process_data(data)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1195, in _process_data
data.reraise()
File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/workspace/monai/monai/data/dataset.py", line 809, in __getitem__
raise RuntimeError(
RuntimeError: if using MONAI workflows, please add `SmartCacheHandler` to the handler list of trainer,otherwise, please make sure to call `start()`, `update_cache()`, `shutdown()` during training.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working