Skip to content

SmartCache is not working in Clara Train #1852

@bhashemian

Description

@bhashemian

Describe the bug
SmartCache is not working when used in an MMAR of Clara Train despite using SmartCacheHandler as the first handler.

To Reproduce
Steps to reproduce the behavior:

  1. Take a Clara Train MMAR
  2. Use SmartCachePatchWSIDataset as the dataset.
  3. Run MMAR

Additional context

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/clara-train/pt/src/medl/apps/train.py", line 35, in <module>
    main()
  File "/workspace/clara-train/pt/src/medl/apps/train.py", line 27, in main
    train_mmar(args)
  File "/workspace/clara-train/pt/src/medl/apps/mmar_conf.py", line 28, in train_mmar
    trainer.run()
  File "/workspace/monai/monai/engines/trainer.py", line 48, in run
    super().run()
  File "/workspace/monai/monai/engines/workflow.py", line 197, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/workspace/monai/monai/handlers/checkpoint_saver.py", line 274, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/workspace/monai/monai/handlers/checkpoint_saver.py", line 274, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
    self.state.batch = next(self._dataloader_iter)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 519, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1169, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1195, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/workspace/monai/monai/data/dataset.py", line 809, in __getitem__
    raise RuntimeError(
RuntimeError: if using MONAI workflows, please add `SmartCacheHandler` to the handler list of trainer,otherwise, please make sure to call `start()`, `update_cache()`, `shutdown()` during training.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions