Skip to content

1014 learning rate finder#1454

Merged
rijobro merged 35 commits intoProject-MONAI:masterfrom
rijobro:learning_rate
Jan 26, 2021
Merged

1014 learning rate finder#1454
rijobro merged 35 commits intoProject-MONAI:masterfrom
rijobro:learning_rate

Conversation

@rijobro
Copy link
Copy Markdown
Contributor

@rijobro rijobro commented Jan 15, 2021 •
edited
Loading

Fixes #1014.

Description

Implements calculation of optimal learning rate based on https://github.com/davidtvs/pytorch-lr-finder.

image

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh --codeformat --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@rijobro
learning rate finder
6f139b2 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
tidying
5c06a4b 
Signed-off-by: Richard Brown <[email protected]>
@Nic-Ma Nic-Ma changed the title learning rate finder 1014 learning rate finder Jan 18, 2021
@Nic-Ma Nic-Ma changed the title 1014 learning rate finder [WIP] 1014 learning rate finder Jan 18, 2021
@rijobro
working
b5c0248 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
autofixes
8ba4230 
Signed-off-by: Richard Brown <[email protected]>
@rijobro rijobro marked this pull request as ready for review January 18, 2021 17:20
rijobro added 12 commits January 19, 2021 09:35
@rijobro
fix type checking problem
cc63970 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
Merge branch 'master' into learning_rate
dd8bdd9
@rijobro
undo typing changes
dbe0409 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
more undo
fdd0c2d 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
try to fix CI/CD
883946c 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
try to fix CI/CD
3925017 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
more fixes
94d0d5e 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
only do unit test if PIL is installed
53a1b38 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
autofix
9ec1beb 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
enhance mednist dataset
3bcdddd 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
Merge branch 'enhance_mednistdataset' into learning_rate
028a965
@rijobro
Merge branch 'master' into learning_rate
4a55bd6
@rijobro rijobro mentioned this pull request Jan 20, 2021
Merged
@rijobro rijobro changed the title [WIP] 1014 learning rate finder 1014 learning rate finder Jan 20, 2021
@rijobro
only test if linux
229fdf0 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
reason
c8d4689 
Signed-off-by: Richard Brown <[email protected]>
@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented Jan 21, 2021

@wyli @Nic-Ma @ericspod this part of the PR saves the state of the network and optimiser to disk or memory, such that they can be restored at the end.

Does this functionality live somewhere else in MONAI?

@ericspod
Copy link
Copy Markdown
Member

ericspod commented Jan 21, 2021

We had that kind of functionality as part of an Ignite handler for saving checkpoints, but that's not quite how you're doing things here.

wyli
wyli reviewed Jan 21, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @rijobro this is very useful!

Perhaps the way the training losses is computed/accumulated (self._train_batch) should be decoupled from the lr_finder. Intuitively I think the LR finder should work fine as long as the user provides a black box that takes lr/step as input, and returns a total_loss (e.g. self._train_batch and self._validate). Do you want to refactor this PR to decouple those? otherwise we could file a ticket and have another iteration.

pls see also some minor suggestions inline, they are mostly optional

@Can-Zhao would be great to have your comments as well!

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented Jan 21, 2021

@wyli thanks, I'll get to it!

@rijobro rijobro mentioned this pull request Jan 22, 2021
Closed
6 tasks
rijobro added 10 commits January 22, 2021 15:47
@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented Jan 25, 2021

@wyli this is ready if you want to review again, thanks!

wyli
wyli approved these changes Jan 26, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, it looks good, except some minor warnings on (object) https://deepsource.io/gh/Project-MONAI/MONAI/run/a53a8313-ab56-4aea-8a8b-e6fd4941f978/python/PYL-R0205 this new feature needs another iteration to decouple the actual training/validation logic_train_batch and _validate from the LearningRateFinder class

@rijobro rijobro merged commit a2da8a1 into Project-MONAI:master Jan 26, 2021
@rijobro rijobro deleted the learning_rate branch January 26, 2021 11:31
@rijobro rijobro mentioned this pull request Jan 26, 2021
Closed
7 tasks
@andre-fs-ferreira
Copy link
Copy Markdown
Contributor

andre-fs-ferreira commented Mar 23, 2021

I am having trouble using this class for GANs. Is there any way to do something similar to the "GANLearner" fastai?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

@ericspod ericspod Awaiting requested review from ericspod

@Nic-Ma Nic-Ma Awaiting requested review from Nic-Ma

@yiheng-wang-nv yiheng-wang-nv Awaiting requested review from yiheng-wang-nv

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

learning rate finder

4 participants

@rijobro @ericspod @andre-fs-ferreira @wyli