Skip to content

Auto3DSeg tests improvement on mGPU ensemble #6456

@mingxin-zheng

Description

@mingxin-zheng

Describe the bug

It runs on single GPU, but fails on 4-GPU.

2023-04-28 13:02:27,188 - INFO - Ensembling using 4 GPU!
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
2023-04-28 13:02:33,112 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,126 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,159 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,272 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str

The cause may be that there are only 2 testing images, and the number of testing images is less than number of GPUs

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions