Skip to content

AutoRunner small updates#5523

Merged
Nic-Ma merged 15 commits intoProject-MONAI:devfrom
myron:autorunner
Nov 22, 2022
Merged

AutoRunner small updates#5523
Nic-Ma merged 15 commits intoProject-MONAI:devfrom
myron:autorunner

Conversation

@myron
Copy link
Copy Markdown
Collaborator

@myron myron commented Nov 15, 2022

AutoRunner() non-breaking class updates, to add more options

  • adding and option self.algos to manually specify algorithms to run (instead of always downloading a zip)
  • Input options train, algo_gen, analyze will default to None, indicating to automatically decide whether this step is necessary based on cache (history).
  • default ensemble method is changed to AlgoEnsembleBestByFold
  • added method set_analyze_params() to set DataAnalyser parameters (e.g. run it on gpu)
  • changed self.cache dict to be a single level (since it only has 4 items), instead of a nested dictionary
  • removed default setting sigmoid=True in set_prediction_params(), since it's somewhat arbitrary

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@myron myron removed the request for review from mingxin-zheng November 17, 2022 08:17
@myron myron removed the request for review from daguangxu November 19, 2022 03:48
@mingxin-zheng
Copy link
Copy Markdown
Contributor

Hi @myron , thanks for creating the MR and I have two minor comments. Everything else looks fine to me.

@myron myron requested a review from mingxin-zheng November 22, 2022 02:21
@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Nov 22, 2022

/build

@Nic-Ma Nic-Ma merged commit 7213060 into Project-MONAI:dev Nov 22, 2022
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Nov 22, 2022

this seems to break the integration tests, shall we revert and double-check?

[2022-11-22T05:54:26.689Z] ======================================================================
[2022-11-22T05:54:26.689Z] ERROR: test_autorunner (__main__.TestAutoRunner)
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "tests/test_integration_autorunner.py", line 108, in test_autorunner
[2022-11-22T05:54:26.689Z]     runner.run()
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/auto_runner.py", line 603, in run
[2022-11-22T05:54:26.689Z]     preds = ensembler(pred_param=self.pred_params)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 156, in __call__
[2022-11-22T05:54:26.689Z]     infer_instance = algo[AlgoEnsembleKeys.ALGO]
[2022-11-22T05:54:26.689Z] TypeError: 'NoneType' object is not subscriptable
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] ======================================================================
[2022-11-22T05:54:26.689Z] ERROR: test_autorunner_hpo (__main__.TestAutoRunner)
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 236, in collect_algos
[2022-11-22T05:54:26.689Z]     algo_id = int(identifier)
[2022-11-22T05:54:26.689Z] ValueError: invalid literal for int() with base 10: '0.0001'
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] The above exception was the direct cause of the following exception:
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "tests/test_integration_autorunner.py", line 138, in test_autorunner_hpo
[2022-11-22T05:54:26.689Z]     runner.run()
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/auto_runner.py", line 600, in run
[2022-11-22T05:54:26.689Z]     builder.set_ensemble_method(self.ensemble_method)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 315, in set_ensemble_method
[2022-11-22T05:54:26.689Z]     ensemble.collect_algos(*args, **kwargs)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 238, in collect_algos
[2022-11-22T05:54:26.689Z]     raise ValueError(f"model identifier {identifier} is not number.") from err
[2022-11-22T05:54:26.689Z] ValueError: model identifier 0.0001 is not number.
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Ran 2 tests in 935.990s

@myron
Copy link
Copy Markdown
Collaborator Author

myron commented Nov 22, 2022

this seems to break the integration tests, shall we revert and double-check?

[2022-11-22T05:54:26.689Z] ======================================================================
[2022-11-22T05:54:26.689Z] ERROR: test_autorunner (__main__.TestAutoRunner)
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "tests/test_integration_autorunner.py", line 108, in test_autorunner
[2022-11-22T05:54:26.689Z]     runner.run()
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/auto_runner.py", line 603, in run
[2022-11-22T05:54:26.689Z]     preds = ensembler(pred_param=self.pred_params)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 156, in __call__
[2022-11-22T05:54:26.689Z]     infer_instance = algo[AlgoEnsembleKeys.ALGO]
[2022-11-22T05:54:26.689Z] TypeError: 'NoneType' object is not subscriptable
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] ======================================================================
[2022-11-22T05:54:26.689Z] ERROR: test_autorunner_hpo (__main__.TestAutoRunner)
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 236, in collect_algos
[2022-11-22T05:54:26.689Z]     algo_id = int(identifier)
[2022-11-22T05:54:26.689Z] ValueError: invalid literal for int() with base 10: '0.0001'
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] The above exception was the direct cause of the following exception:
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] Traceback (most recent call last):
[2022-11-22T05:54:26.689Z]   File "tests/test_integration_autorunner.py", line 138, in test_autorunner_hpo
[2022-11-22T05:54:26.689Z]     runner.run()
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/auto_runner.py", line 600, in run
[2022-11-22T05:54:26.689Z]     builder.set_ensemble_method(self.ensemble_method)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 315, in set_ensemble_method
[2022-11-22T05:54:26.689Z]     ensemble.collect_algos(*args, **kwargs)
[2022-11-22T05:54:26.689Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/monai/apps/auto3dseg/ensemble_builder.py", line 238, in collect_algos
[2022-11-22T05:54:26.689Z]     raise ValueError(f"model identifier {identifier} is not number.") from err
[2022-11-22T05:54:26.689Z] ValueError: model identifier 0.0001 is not number.
[2022-11-22T05:54:26.689Z] 
[2022-11-22T05:54:26.689Z] ----------------------------------------------------------------------
[2022-11-22T05:54:26.689Z] Ran 2 tests in 935.990s

@wyli can I ask where are these integration tests and how to run it? They all seems to pass on my local machine and during PR submission.
There will be more changes to Autorunner and auto3dseg coming, including adding/changing more options to auto-runner. So I'd like to run these tests in the future, to detect it early. thank you

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Nov 22, 2022

yes, mainly these commands:

BUILD_MONAI=1 ./runtests.sh --build --coverage --unittests --disttests # unit tests with coverage report
BUILD_MONAI=1 ./runtests.sh --build --coverage --net # integration tests with coverage report

there are some more details in the contributing guide: https://github.com/Project-MONAI/MONAI/blob/dev/CONTRIBUTING.md

wyli added a commit that referenced this pull request Nov 22, 2022
This was referenced Nov 22, 2022
bhashemian pushed a commit to bhashemian/MONAI that referenced this pull request Nov 23, 2022
AutoRunner() non-breaking class updates, to add more options 

- adding and option self.algos to manually specify algorithms to run
(instead of always downloading a zip)
- Input options train, algo_gen, analyze will default to None,
indicating to automatically decide whether this step is necessary based
on cache (history).
- default ensemble method is changed to AlgoEnsembleBestByFold
- added method set_analyze_params() to set DataAnalyser parameters (e.g.
run it on gpu)
- changed self.cache dict to be a single level (since it only has 4
items), instead of a nested dictionary
- removed default setting sigmoid=True in set_prediction_params(), since
it's somewhat arbitrary

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: myron <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Behrooz <[email protected]>
bhashemian pushed a commit to bhashemian/MONAI that referenced this pull request Nov 23, 2022
bhashemian pushed a commit to bhashemian/MONAI that referenced this pull request Nov 23, 2022
AutoRunner() non-breaking class updates, to add more options 

- adding and option self.algos to manually specify algorithms to run
(instead of always downloading a zip)
- Input options train, algo_gen, analyze will default to None,
indicating to automatically decide whether this step is necessary based
on cache (history).
- default ensemble method is changed to AlgoEnsembleBestByFold
- added method set_analyze_params() to set DataAnalyser parameters (e.g.
run it on gpu)
- changed self.cache dict to be a single level (since it only has 4
items), instead of a nested dictionary
- removed default setting sigmoid=True in set_prediction_params(), since
it's somewhat arbitrary

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: myron <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Behrooz <[email protected]>
bhashemian pushed a commit to bhashemian/MONAI that referenced this pull request Nov 23, 2022
wyli pushed a commit that referenced this pull request Nov 23, 2022
This is the same PR as in
#5523

but removing changes to default ensemble method
(AlgoEnsembleBestByFold), and keeping it as is for now, since it triggers
errors with some unit tests. the default ensemble method update can be
done in a separate PR.


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: myron <[email protected]>
wyli pushed a commit to bhashemian/MONAI that referenced this pull request Nov 23, 2022
This is the same PR as in
Project-MONAI#5523

but removing changes to default ensemble method
(AlgoEnsembleBestByFold), and keeping it as is for now, since it triggers
errors with some unit tests. the default ensemble method update can be
done in a separate PR.


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: myron <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants