Redefine scheduler to set learning rate using recursive formula #14010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

chandlerzuo wants to merge 5 commits into pytorch:master from chandlerzuo:master

Contributor

chandlerzuo commented Nov 15, 2018 •

edited

Loading

Modified step_lr for StepLR, MultiStepLR, ExponentialLR and CosineAnnealingLR. In this way, multiple schedulers can be used simultaneously to modify the learning rates.

Related issue: #13022

Added unit tests combining multiple schedulers.


          Fix issue pytorch#13022

7d14f18

ezyang reviewed

View reviewed changes

torch/optim/lr_scheduler.py Outdated

    
                      return [group['lr'] * self.gamma

                              for group in self.optimizer.param_groups]

                      # return [base_lr * self.gamma ** (self.last_epoch // self.step_size)

                      #         for base_lr in self.base_lrs]

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

ezyang reviewed

View reviewed changes

test/test_optim.py

    
                                                    threshold=0.1, patience=5, cooldown=5)

                      self._test_reduce_lr_on_plateau(scheduler, targets, metrics, epochs)

                  def test_compound_step_and_multistep_lr(self):

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Contributor

ezyang commented Nov 15, 2018

Don't forget to fix Python lint (see the Travis job.)

ezyang reviewed

View reviewed changes

torch/optim/lr_scheduler.py

    
                  def get_lr(self):

                      return [base_lr * self.gamma ** (self.last_epoch // self.step_size)

                              for base_lr in self.base_lrs]

                      if (self.last_epoch == 0) or (self.last_epoch % self.step_size != 0):

This comment was marked as off-topic.

Sign in to view

ezyang reviewed

View reviewed changes

torch/optim/lr_scheduler.py Outdated

    
                      milestones = set(milestones)

                      #if not list(milestones) == sorted(milestones):

                      #    raise ValueError('Milestones should be a list of'

                      #                     ' increasing integers. Got {}', milestones)

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Contributor

ezyang commented Nov 15, 2018

Changes LGTM.

Contributor

ezyang commented Nov 15, 2018

Nit: When naming a PR, name it for what it does, not a URL (that you have to click through to find out what it's about ;)

chandlerzuo added 2 commits

November 15, 2018 10:09


          cleaned codes, added BC tests

0e58975


          cosine formula

9fcfa3b

chandlerzuo changed the title ~~Fix issue https://github.com/pytorch/pytorch/issues/13022~~ Redefine scheduler to set learning rate using recursive formula


          Fixed flake8 errors

bda21bd

ezyang approved these changes

View reviewed changes

jeanm commented Dec 15, 2018

Is this blocked on something? Would be great to have it merged!

facebook-github-bot reviewed

View reviewed changes

Contributor

facebook-github-bot left a comment

@chandlerzuo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot reviewed

View reviewed changes

Contributor

facebook-github-bot left a comment

@chandlerzuo is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot reviewed

View reviewed changes

Contributor

facebook-github-bot left a comment

@chandlerzuo is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Merge remote-tracking branch 'upstream/master'

392e380

facebook-github-bot closed this in

096ee84

striajan mentioned this pull request

Cosine annealing with restarts #11104

Closed

ptrblck mentioned this pull request

Bug in CosineAnnealingLR (division by zero) #17913

Closed

fmassa mentioned this pull request

Support multiple simultaneous LR schedulers #13022

Closed

ezyang added a commit that referenced this pull request


          Revert "Redefine scheduler to set learning rate using recursive formu…

a2d58a5

…la (#14010)"

This reverts commit 096ee84.

See #17913

ezyang mentioned this pull request

Revert "Redefine scheduler to set learning rate using recursive formula (#14010)" #19132

Closed

ezyang added a commit that referenced this pull request


          Revert "Redefine scheduler to set learning rate using recursive formu…

e75f00d

…la (#14010)"

This reverts commit 096ee84.

See #17913

gh-metadata: pytorch pytorch 19132 gh/ezyang/69/head

vfdev-5 mentioned this pull request

Fix CI pytorch/ignite#503

Closed

vfdev-5 mentioned this pull request

Inconsistant values of lr_scheduler.get_lr and lr in optimizer.param_groups #20138

Open

jasam-sheja mentioned this pull request

StepLR, MultiStepLR, ExponentialLR and CosineAnnealingLR scheduler wrong lr value #20527

Closed

ezyang added a commit that referenced this pull request


          Revert "Redefine scheduler to set learning rate using recursive formu…

deadc2a

…la (#14010)"

This reverts commit 096ee84.  It also
reverts some subsequent fixes.

Fixes #20527

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

9ab9ad5

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

f7f0d08

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

6f79011

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

f74522b

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter WIP"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

b59ab67

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

fbb71a0

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

88f2f25

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

8c5795b

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

d8f2bff

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

e6ce88e

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

af35fdb

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

8e60db3

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

ezyang added the merged label

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

1d4d394

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

fde1c28

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

197c8dc

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

40fd68d

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

fe4d9fe

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

8bc446e

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

f9301e2

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

388bb23

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

6fca527

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Update on "Revert "Revert "Redefine scheduler to set learning rate us…

364d24c

…ing recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter"

Revert "Revert "Redefine scheduler to set learning rate using recursive formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

a551b97

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

gh-metadata: pytorch pytorch 21800 gh/vincentqb/6/head

vincentqb added a commit that referenced this pull request


          Revert "Revert "Redefine scheduler to set learning rate using recursi…

5a7b329

…ve formula" #14010 (#21463)" and enable closed form with non-sequential epoch parameter

This reverts commit 3889855.

vincentqb mentioned this pull request

[WIP] Create new ChainLR. #22798

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet