Skip to content

Fixed behavior of div_factor parameter in optim.lr_scheduler.OneCycleLR #28217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
timothyman wants to merge 4 commits into from
Closed

Fixed behavior of div_factor parameter in optim.lr_scheduler.OneCycleLR #28217

timothyman wants to merge 4 commits into from

Conversation

@timothyman
Copy link

@timothyman timothyman commented Oct 17, 2019

Fixes #28216

@smessmer smessmer requested a review from fmassa October 17, 2019 19:35
@smessmer
Copy link
Contributor

smessmer commented Oct 17, 2019

@fmassa can you take a look at this?

@soumith soumith requested a review from vincentqb October 17, 2019 22:12
@timothyman timothyman force-pushed the div_factor_in_onecyclelr branch from 23c3f80 to 6dfd735 Compare October 18, 2019 07:16
vincentqb
vincentqb suggested changes Oct 18, 2019
View reviewed changes
Copy link
Contributor

@vincentqb vincentqb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test to show that #28216 is indeed fixed? I tried

    def test_onecycle_lr_div_factor(self):                          
         model = torch.nn.Linear(2, 1)
         optimizer = optim.Adam(model.parameters(), lr=1E-2)
         lr_scheduler = OneCycleLR(optimizer,
                 max_lr=1E-2,
                 total_steps=100,
                 cycle_momentum=False,
                 div_factor=25)

         # Plot the Learning Rate
         lr1 = []
         for epoch in range(100):
             optimizer.zero_grad()
             optimizer.step()
             lr_scheduler.step()
             for param_group in optimizer.param_groups:
                 lr1.append(param_group['lr'])

         # Change the div_factor and plot again (step 2).
         model = torch.nn.Linear(2, 1)
         optimizer = optim.Adam(model.parameters(), lr=1E-2)
         lr_scheduler = OneCycleLR(optimizer,
                 max_lr=1E-2,
                 total_steps=100,
                 cycle_momentum=False,
                 div_factor=10000)

         # Plot the Learning Rate
         lr2 = []
         for epoch in range(100):
             optimizer.zero_grad()
             optimizer.step()
             lr_scheduler.step()
             for param_group in optimizer.param_groups:
                 lr2.append(param_group['lr'])

         print(lr1)
         print(lr2)

and I get

[0.00042813780565907275, 0.000512221332591583, 0.0006512647783225474, 0.0008436379855794058, 0.001087085554395571, 0.0013787532846129096, 0.0017152216387689712, 0.002092545833047226, 0.002506302086260563, 0.002951639484641007, 0.0034233368543684093, 0.003915863975059738, 0.004423446416546728, 0.004940133238789995, 0.005459866761210004, 0.005976553583453271, 0.006484136024940259, 0.006976663145631589, 0.007448360515358993, 0.007893697913739435, 0.008307454166952773, 0.008684778361231027, 0.00902124671538709, 0.009312914445604427, 0.009556362014420594, 0.009748735221677452, 0.009887778667408417, 0.009971862194340927, 0.01, 0.009994965352845242, 0.009979871550490316, 0.009954748989844438, 0.009919648264401377, 0.009874640062350875, 0.009819815024222052, 0.009755283560345442, 0.009681175628501274, 0.009597640472201803, 0.009504846320134737, 0.009402980047373053, 0.009292246799033458, 0.009172869577141439, 0.00904508879153485, 0.0089091617757105, 0.008765362268588735, 0.008613979863239682, 0.008455319423681345, 0.008289700470924045, 0.008117456539497631, 0.007938934505757321, 0.007754493889320884, 0.007564506129043983, 0.007369353834991744, 0.007169430017913008, 0.006965137297768985, 0.006756887092910232, 0.00654509879153485, 0.006330198907096464, 0.0061126202193628925, 0.005892800902855287, 0.005671183644422961, 0.00544821475173099, 0.005224343254455968, 0.00500002, 0.004775696745544034, 0.004551825248269011, 0.004328856355577041, 0.004107239097144713, 0.003887419780637108, 0.0036698410929035365, 0.003454941208465151, 0.003243152907089768, 0.003034902702231016, 0.0028306099820869924, 0.0026306861650082568, 0.0024355338709560184, 0.002245546110679117, 0.002061105494242681, 0.0018825834605023704, 0.0017103395290759564, 0.0015447205763186572, 0.0013860601367603187, 0.0012346777314112658, 0.001090878224289501, 0.0009549512084651508, 0.0008271704228585612, 0.0007077932009665415, 0.0005970599526269471, 0.0004951936798652628, 0.00040239952779819817, 0.0003188643714987268, 0.00024475643965455826, 0.00018022497577794772, 0.00012539993764912555, 8.039173559862365e-05, 4.529101015556186e-05, 2.0168449509684565e-05, 5.07464715475788e-06, 4e-08, 5.074647154757324e-06]

[3.0307283206777724e-05, 0.00011788553172742064, 0.00026270797067157875, 0.0004630766893550485, 0.0007166425477501369, 0.0010204327180046331, 0.0013708855381303064, 0.0017638922692332516, 0.002194845266720768, 0.0026586920007213985, 0.0031499942923780963, 0.0036629920715231584, 0.004191670908234452, 0.0047298325265272044, 0.005271167473472794, 0.005809329091765547, 0.0063380079284768385, 0.006851005707621902, 0.0073423079992786, 0.00780615473327923, 0.008237107730766746, 0.008630114461869691, 0.008980567281995366, 0.00928435745224986, 0.00953792331064495, 0.00973829202932842, 0.009883114468272579, 0.009970692716793222, 0.01, 0.00999496533275692, 0.009979871470177482, 0.009954748809292186, 0.00991964794379667, 0.009874639562162722, 0.009819814305281123, 0.009755282583922942, 0.009681174356386943, 0.009597638866780866, 0.009504844344463652, 0.009402977665253914, 0.009292243975086892, 0.00917286627687785, 0.009045084981423888, 0.008909157423248576, 0.008765357342364482, 0.008613974332997215, 0.008455313260381191, 0.008289693646801629, 0.008117449028119178, 0.007938926282073105, 0.007754484929715664, 0.0075644964113845675, 0.00736934333867156, 0.007169418723893603, 0.0069651251886183665, 0.006756874152837973, 0.006545085006423887, 0.006330184264531533, 0.006112604708655525, 0.005892784515065338, 0.005671166372376614, 0.005448196590035202, 0.005224324199509333, 0.00500000005, 0.004775675900490668, 0.004551803509964798, 0.004328833727623388, 0.0041072155849346615, 0.003887395391344475, 0.0036698158354684673, 0.0034549150935761132, 0.0032431259471620275, 0.003034874911381634, 0.002830581376106397, 0.0026306567613284403, 0.002435503688615434, 0.002245515170284337, 0.0020610738179268977, 0.0018825510718808234, 0.001710306453198374, 0.00154468683961881, 0.0013860257670027854, 0.0012346427576355192, 0.0010908426767514258, 0.0009549151185761131, 0.0008271338231221495, 0.0007077561249131093, 0.0005970224347460867, 0.0004951557555363483, 0.00040236123321913574, 0.00031882574361305753, 0.00024471751607705813, 0.00018018579471887686, 0.00012536053783727751, 8.035215620333109e-05, 4.525129070781459e-05, 2.0128629822519358e-05, 5.03476724308038e-06, 9.999999999999999e-11, 5.034767243079826e-06]

Is that what's expected?

@timothyman timothyman requested a review from vincentqb October 21, 2019 09:09
@timothyman
Copy link
Author

timothyman commented Oct 21, 2019

I added a unit-test with the expected behavior.

@timothyman timothyman mentioned this pull request Oct 21, 2019
Closed
@timothyman
Copy link
Author

timothyman commented Oct 21, 2019 •
edited
Loading

Is that what's expected?

I added another comment on #28216 . It looks like your example is indeed what's expected, even in 1.3.0, i.e. it behaves correctly as long as you pass a new optimizer in the new OneCycleLR. However, when optimizers are reused, it shows the behavior I logged in the issue.

vincentqb
vincentqb reviewed Oct 21, 2019
View reviewed changes
vincentqb
vincentqb approved these changes Oct 22, 2019
View reviewed changes
Copy link
Contributor

@vincentqb vincentqb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

facebook-github-bot
facebook-github-bot reviewed Oct 22, 2019
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vincentqb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@timothyman timothyman deleted the div_factor_in_onecyclelr branch October 23, 2019 20:43
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Oct 23, 2019

@vincentqb merged this pull request in 1c53a74.

thiagocrepaldi pushed a commit to thiagocrepaldi/pytorch that referenced this pull request Feb 4, 2020
Fixed behavior of div_factor parameter in optim.lr_scheduler.OneCycle…
bf2f9f3 
…LR (pytorch#28217)

Summary:
Fixes pytorch#28216
Pull Request resolved: pytorch#28217

Differential Revision: D18070759

Pulled By: vincentqb

fbshipit-source-id: ed032190c0e3eab834fc9a8f408b75b56f0f35ec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@fmassa fmassa Awaiting requested review from fmassa

+1 more reviewer

@vincentqb vincentqb vincentqb approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The div_factor parameter in OneCycleLR does not work

5 participants

@timothyman @smessmer @facebook-github-bot @vincentqb @mruberry