Skip to content

5385-enhance-mlflow-handler#5388

Merged
wyli merged 63 commits intoProject-MONAI:devfrom
binliunls:5385-enhance-mlflow-handler
Dec 2, 2022
Merged

5385-enhance-mlflow-handler#5388
wyli merged 63 commits intoProject-MONAI:devfrom
binliunls:5385-enhance-mlflow-handler

Conversation

@binliunls
Copy link
Copy Markdown
Contributor

@binliunls binliunls commented Oct 24, 2022 •
edited
Loading

Signed-off-by: binliu [email protected]

Fixes #5385 .

Description

This PR is about to enhance the mlflow handler in monai to track more details in the experiment. Here are a few enhancements
that needs to be added through this PR.

  • API for users to add experiment/run name in MLFlow
  • API for users to log customized params for each run
  • Methods to log result images
  • Methods to log optimizer params
  • (optional) additional metric_names as a user argument to override the default engine.state.metrics to instruct MLFlow about metrics to log

After adding these enhancements, some tests listed below should be excuted.

  • Make sure this handler works in multi-gpu environment
  • Make sure this handler works in all existed bundles

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.
  • API for users to add experiment/run name in MLFlow
  • API for users to log customized params for each run
  • Methods to log result images
  • Methods to log optimizer params
  • Make sure this handler works in multi-gpu environment
  • Make sure this handler works in all existed bundles

mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
mingxin-zheng
mingxin-zheng reviewed Oct 28, 2022
View reviewed changes
Nic-Ma
Nic-Ma reviewed Nov 3, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@Nic-Ma Nic-Ma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, put some comments inline.
Need @wyli @ericspod @vfdev-5 for more design discussion.

Thanks in advance.

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Nov 11, 2022 •
edited
Loading

Please ensure we can call these get_config_values() and get_optional_config_values() functions to track the system env info in MLFlow:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/config/deviceconfig.py#L48-L81

CC @wyli

Thanks.

binliunls and others added 12 commits November 15, 2022 14:34
@binliunls
merge dev branch
736053b 
Signed-off-by: binliu <[email protected]>
@binliunls
change mlflow run name according to input parameter.
34e3a8a 
Signed-off-by: binliu <[email protected]>
@binliunls
merge dev branch to current branch
8325cf3 
Signed-off-by: binliu <[email protected]>
@binliunls
format code
a92eb19 
Signed-off-by: binliu <[email protected]>
@binliunls
Merge branch 'Project-MONAI:dev' into 5385-enhance-mlflow-handler
765061b
@binliunls
fix codeformat problem
95227f7 
Signed-off-by: binliu <[email protected]>
@binliunls
Merge branch '5385-enhance-mlflow-handler' of https://github.com/binl…
6ee0dbf 
…iunls/MONAI into 5385-enhance-mlflow-handler
@binliunls
update the code comments and handler for validator
5d4475f 
Signed-off-by: binliu <[email protected]>
@binliunls
update some type hints and comments
e8760d7 
Signed-off-by: binliu <[email protected]>
@binliunls
update default mlflow tracking with bundle info and multi gpu info
7fe8269 
Signed-off-by: binliu <[email protected]>
@binliunls
fix the log default parameter bug
39f40b6 
Signed-off-by: binliu <[email protected]>
@binliunls
add disable for multi gpu in validator and evaluator
e3d2cb3 
Signed-off-by: binliu <[email protected]>
Nic-Ma
Nic-Ma reviewed Dec 1, 2022
View reviewed changes
Nic-Ma
Nic-Ma reviewed Dec 1, 2022
View reviewed changes
binliunls and others added 7 commits December 1, 2022 07:35
wyli
wyli reviewed Dec 1, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I put some minor comments inline

wyli
wyli approved these changes Dec 1, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, @Nic-Ma pls let me know if you still have comments otherwise I'll merge this later today

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Dec 1, 2022

/build

Nic-Ma
Nic-Ma requested changes Dec 1, 2022
View reviewed changes
@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Dec 1, 2022

Could you please help also add the experiment_name and run_name in this doc-string example:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/bundle/scripts.py#L549

Thanks.

Nic-Ma
Nic-Ma reviewed Dec 2, 2022
View reviewed changes
@Nic-Ma Nic-Ma requested a review from wyli December 2, 2022 15:47
@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Dec 2, 2022

/build

@wyli wyli enabled auto-merge (squash) December 2, 2022 16:07
@wyli wyli merged commit 7b41e2e into Project-MONAI:dev Dec 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mingxin-zheng mingxin-zheng mingxin-zheng left review comments

@Nic-Ma Nic-Ma Nic-Ma approved these changes

@wyli wyli Awaiting requested review from wyli

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Enhance MLFlowHandler

4 participants

@binliunls @Nic-Ma @wyli @mingxin-zheng