-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Auto3DSeg tests improvement on mGPU ensemble #6456
Copy link
Copy link
Closed
Description
Describe the bug
It runs on single GPU, but fails on 4-GPU.
2023-04-28 13:02:27,188 - INFO - Ensembling using 4 GPU!
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
2023-04-28 13:02:33,112 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,126 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,159 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,272 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
fire.Fire(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
self.ensembler(pred_param=self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
fire.Fire(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
self.ensembler(pred_param=self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
fire.Fire(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
self.ensembler(pred_param=self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
fire.Fire(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
self.ensembler(pred_param=self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
files = files[slices]
TypeError: list indices must be integers or slices, not str
The cause may be that there are only 2 testing images, and the number of testing images is less than number of GPUs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels