AutoRunner enhancements: local folder and algorithm subsets by myron · Pull Request #5623 · Project-MONAI/MONAI

myron · 2022-12-01T11:02:25Z

AutoRunner class improvements

adds auto-runner option templates_path_or_url to optionally accept a local templates folder location. The default (None) is the same: to download the release zip e.g. AutoRunner(templates_path_or_url='/my/local/algorithm_templates')
enhances algos input option to define a subset of algorithm names to run (e.g. AutoRunner(algos=['segresnet','dints'])

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: myron <[email protected]>

for more information, see https://pre-commit.ci

myron · 2022-12-01T11:25:37Z

the windows tests are failing again , for some unrelated reason...

Nic-Ma · 2022-12-01T14:29:27Z

I see the flake8 tests didn't pass, could you please help fix it?

Thanks.

mingxin-zheng · 2022-12-01T15:18:13Z

I checked out the branch and run the integration test for AutoRunner. Some failures need to be fixed:

(base) root@adminn-MS-7D31:/workspace/monai/monai-in-dev# python tests/test_integration_autorunner.py
2022-12-01 15:16:54,155 - INFO - AutoRunner using work directory /tmp/tmplf3777to/work_dir
2022-12-01 15:16:54,156 - INFO - Loading input config /tmp/tmplf3777to/data_src_cfg.yaml
2022-12-01 15:16:54,157 - INFO - The output_dir is not specified. /tmp/tmplf3777to/work_dir/ensemble_output will be used to save ensemble predictions
2022-12-01 15:16:54,157 - INFO - Directory /tmp/tmplf3777to/work_dir/ensemble_output is created to save ensemble predictions
2022-12-01 15:16:54,157 - INFO - Running data analysis...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 45.86it/s]
algo_templates.tar.gz: 296kB [00:01, 243kB/s]                                                                                                                                          
2022-12-01 15:16:55,743 - INFO - Downloaded: /tmp/tmpphes5ts9/algo_templates.tar.gz
2022-12-01 15:16:55,744 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmpphes5ts9/algo_templates.tar.gz.
2022-12-01 15:16:55,744 - INFO - Writing into directory: /tmp/tmplf3777to/work_dir.
E2022-12-01 15:16:55,897 - INFO - AutoRunner using work directory /tmp/tmpvo558ukw/work_dir
2022-12-01 15:16:55,898 - INFO - Loading input config /tmp/tmpvo558ukw/data_src_cfg.yaml
2022-12-01 15:16:55,898 - INFO - The output_dir is not specified. /tmp/tmpvo558ukw/work_dir/ensemble_output will be used to save ensemble predictions
2022-12-01 15:16:55,898 - INFO - Directory /tmp/tmpvo558ukw/work_dir/ensemble_output is created to save ensemble predictions
2022-12-01 15:16:55,898 - INFO - Running data analysis...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 75.03it/s]
algo_templates.tar.gz: 296kB [00:01, 298kB/s]                                                                                                                                          
2022-12-01 15:16:57,142 - INFO - Downloaded: /tmp/tmp2m08_yi6/algo_templates.tar.gz
2022-12-01 15:16:57,142 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmp2m08_yi6/algo_templates.tar.gz.
2022-12-01 15:16:57,143 - INFO - Writing into directory: /tmp/tmpvo558ukw/work_dir.
E
======================================================================
ERROR: test_autorunner (__main__.TestAutoRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_integration_autorunner.py", line 108, in test_autorunner
    runner.run()
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/auto_runner.py", line 566, in run
    bundle_generator = BundleGen(
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/bundle_gen.py", line 349, in __init__
    for name in algos:
TypeError: 'NoneType' object is not iterable

======================================================================
ERROR: test_autorunner_hpo (__main__.TestAutoRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_integration_autorunner.py", line 138, in test_autorunner_hpo
    runner.run()
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/auto_runner.py", line 566, in run
    bundle_generator = BundleGen(
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/bundle_gen.py", line 349, in __init__
    for name in algos:
TypeError: 'NoneType' object is not iterable

----------------------------------------------------------------------
Ran 2 tests in 3.141s

FAILED (errors=2)

monai/apps/auto3dseg/bundle_gen.py

monai/apps/auto3dseg/auto_runner.py

Signed-off-by: myron <[email protected]>

monai/apps/auto3dseg/bundle_gen.py

Signed-off-by: myron <[email protected]>

monai/apps/auto3dseg/bundle_gen.py

Signed-off-by: myron <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: myron <[email protected]>

myron · 2022-12-02T02:31:06Z

I see the flake8 tests didn't pass, could you please help fix it?

Thanks.

done

myron · 2022-12-02T02:31:31Z

I checked out the branch and run the integration test for AutoRunner. Some failures need to be fixed:

done

Signed-off-by: myron <[email protected]>

wyli

thanks, I put some some comments inline

monai/apps/auto3dseg/auto_runner.py

monai/apps/auto3dseg/bundle_gen.py

Signed-off-by: myron <[email protected]>

for more information, see https://pre-commit.ci

myron · 2022-12-02T20:36:30Z

@wyli @mingxin-zheng I've made the requested changes, plz take a look.

wyli

looks good to me, please confirm @mingxin-zheng

monai/apps/auto3dseg/bundle_gen.py

mingxin-zheng · 2022-12-05T07:27:02Z

Hi @myron , I just had a small comment and everything else looked good! Thx

wyli · 2022-12-05T17:34:17Z

/build

adding local folder option

2eca6b2

Signed-off-by: myron <[email protected]>

myron added the enhancement New feature or request label Dec 1, 2022

myron added this to the Auto3D Seg framework [internal ongoing milestone] milestone Dec 1, 2022

[pre-commit.ci] auto fixes from pre-commit.com hooks

bd0c552

for more information, see https://pre-commit.ci

myron requested a review from mingxin-zheng December 1, 2022 11:03

mingxin-zheng requested a review from wyli December 1, 2022 14:04

wyli requested a review from Nic-Ma December 1, 2022 14:06