feat: Added warmup each cycle feature in CyclicalScheduler by sihyeong671 · Pull Request #3064 · pytorch/ignite

sihyeong671 · 2023-09-16T05:27:29Z

add _get_param method in CyclicalScheduler
add warmup_duration variable in CyclicalScheduler

Description:
this feature is following https://github.com/katsura-jp/pytorch-cosine-annealing-with-warmup

import numpy as np
import matplotlib.pylab as plt
from ignite.handlers import CosineAnnealingScheduler


lr_values_1 = np.array(CosineAnnealingScheduler.simulate_values(
    num_events=300,
    param_name='lr',
    start_value=1.0, 
    end_value=0.0,
    cycle_size=100,
    warmup_duration=25,)
)


plt.figure(figsize=(12, 9))
plt.title("LinearCyclicalScheduler")
plt.plot(lr_values_1[:, 0], lr_values_1[:, 1], label="learning rate")
plt.xlabel("events")
plt.ylabel("values")
plt.legend()
plt.ylim([0.0, 1.1])
plt.show()

Check list:

New tests are added (if a new feature is added)
New doc strings: description and/or example code are in RST format
Documentation is updated (if required)

- add _get_cycle_param method in CyclicalScheduler - add warmup_each_cycle, warmup_duration variable in CyclicalScheduler - add warmup phase in CosineAnnealingScheduler issue pytorch#3036

ignite/handlers/param_scheduler.py

- remove f in not using f string variable sentence

- rename get_param in LinearCyclicalScheduler, CosineAnnealingScheduler to _get_clcye_param

sadra-barikbin

Thank you @sihyeong671 , I left some comments.

ignite/handlers/param_scheduler.py

- fix typo - add docsting which is in PR review - change _get_cycle_param to abstractmethod - raise ValueError when warmup_each_cycle=True && warmup_duration is None

- add test_cyclical_scheduler_asserts - add test_cosine_annealing_scheduler_warmup

vfdev-5 · 2023-09-18T07:25:04Z

@sihyeong671 thanks for working on this PR! Can you please fix this test issue:

FAILED tests/ignite/handlers/test_param_scheduler.py::test_cyclical_scheduler_asserts - AssertionError: Regex pattern did not match.
 Regex: "Can't instantiate abstract class CyclicalScheduler with abstract method _get_cycle_param"
 Input: "Can't instantiate abstract class CyclicalScheduler with abstract methods _get_cycle_param"

ignite/handlers/param_scheduler.py

sihyeong671 · 2023-09-18T07:49:36Z

@sihyeong671 thanks for working on this PR! Can you please fix this test issue:

FAILED tests/ignite/handlers/test_param_scheduler.py::test_cyclical_scheduler_asserts - AssertionError: Regex pattern did not match.
 Regex: "Can't instantiate abstract class CyclicalScheduler with abstract method _get_cycle_param"
 Input: "Can't instantiate abstract class CyclicalScheduler with abstract methods _get_cycle_param"

In macOS, If I change the sentence, AssertionError like below code
First, I will change it and PR

E           AssertionError: Regex pattern did not match.
E            Regex: "Can't instantiate abstract class CyclicalScheduler with abstract methods _get_cycle_param"
E            Input: "Can't instantiate abstract class CyclicalScheduler with abstract method _get_cycle_param"

vfdev-5 · 2023-09-18T07:55:32Z

@sihyeong671 I see, let's just check for this sting "Can't instantiate abstract class CyclicalScheduler":

    with pytest.raises(
        TypeError, match="Can't instantiate abstract class CyclicalScheduler"
    ):
        CyclicalScheduler({}, "lr", 0.0, 1.0, 10)

sihyeong671 · 2023-09-18T09:21:41Z

@vfdev-5 I have no idea how to fix the error. In cpu-tests, error shows "Error: The operation was canceled.", how to handle it?

vfdev-5 · 2023-09-18T09:24:22Z

@sihyeong671 no worries, this can be a flaky CI. We set 45 minutes limit if the test stuck with distributed tests. So, just restarting it.

ignite/handlers/param_scheduler.py

- change _get_cycle_param to _get_param - add _get_param in ParamScheduler - remove warmup_each_cycle variable

- remove test_cyclical_scheduler - remove warmup_each_cycle variable

ignite/handlers/param_scheduler.py

vfdev-5 · 2023-09-19T10:41:07Z

@sihyeong671 can you please fix mypy issue: https://github.com/pytorch/ignite/actions/runs/6231400451/job/16912898300?pr=3064#step:12:18

…feat/cycle_scheduler

ignite/handlers/param_scheduler.py

vfdev-5

LGTM, thanks @sihyeong671 and @sadra-barikbin !

feat: add feature each cycle in cosine annealing

731a943

- add _get_cycle_param method in CyclicalScheduler - add warmup_each_cycle, warmup_duration variable in CyclicalScheduler - add warmup phase in CosineAnnealingScheduler issue pytorch#3036

github-actions bot added the module: handlers Core Handlers module label Sep 16, 2023

vfdev-5 requested a review from sadra-barikbin September 16, 2023 15:11

vfdev-5 reviewed Sep 16, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

sihyeong671 added 3 commits September 17, 2023 16:37

fix: f string

7f2d62c

- remove f in not using f string variable sentence

refactor: add _get_cycle_param, get_param in CyclicalScheduler

aab440d

- rename get_param in LinearCyclicalScheduler, CosineAnnealingScheduler to _get_clcye_param

fix: add total_cycle_size in _state_attrs

ca3033a

sihyeong671 requested a review from vfdev-5 September 17, 2023 08:27

sadra-barikbin reviewed Sep 17, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

sadra-barikbin reviewed Sep 17, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

sihyeong671 added 2 commits September 18, 2023 11:26

fix: add docstring, change function to abstractmethod

7847de3

- fix typo - add docsting which is in PR review - change _get_cycle_param to abstractmethod - raise ValueError when warmup_each_cycle=True && warmup_duration is None

feat: add test function

a4456cf

- add test_cyclical_scheduler_asserts - add test_cosine_annealing_scheduler_warmup

sihyeong671 requested a review from sadra-barikbin September 18, 2023 02:46

vfdev-5 reviewed Sep 18, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

vfdev-5 reviewed Sep 18, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

fix: keep previous tag, fix test sentence

e23f746

fix: type error match sentence

b5d358a

sihyeong671 requested a review from vfdev-5 September 18, 2023 09:19

vfdev-5 reviewed Sep 18, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

sihyeong671 added 3 commits September 19, 2023 13:51

feat: refactor _get_cycle_param to _get_param

fdd5512

- change _get_cycle_param to _get_param - add _get_param in ParamScheduler - remove warmup_each_cycle variable

docs: back docstring

bbda3fe

feat: remove & fix

22f5721

- remove test_cyclical_scheduler - remove warmup_each_cycle variable

vfdev-5 reviewed Sep 19, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

sihyeong671 and others added 4 commits September 24, 2023 01:27

feat: remove first cycle warmup

9a71fa9

feat: fix lrs value

323a866

A few improvements

66e5993

Merge remote-tracking branch 'sihyeong671/feat/cycle_scheduler' into …

9012d0c

…feat/cycle_scheduler

sadra-barikbin requested a review from vfdev-5 September 23, 2023 20:49

vfdev-5 reviewed Sep 24, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

vfdev-5 reviewed Sep 24, 2023

View reviewed changes

ignite/handlers/param_scheduler.py Outdated Show resolved Hide resolved

Update param_scheduler.py

c02a78c

vfdev-5 approved these changes Sep 24, 2023

View reviewed changes

vfdev-5 added 2 commits September 25, 2023 00:07

Merge branch 'master' into feat/cycle_scheduler

d588a97

Update test_param_scheduler.py

abb4099

sadra-barikbin approved these changes Sep 24, 2023

View reviewed changes

vfdev-5 merged commit 169551e into pytorch:master Sep 24, 2023

vfdev-5 mentioned this pull request Oct 2, 2023

Is it good to Add Feature "cosine annealing with warmup restart each cycle"? #3036

Closed

sihyeong671 mentioned this pull request Jan 11, 2024

Changing LinearCyclicalScheduler triangle wave to sawtooth wave #3185

Closed

Uh oh!

Conversation

sihyeong671 commented Sep 16, 2023 • edited by vfdev-5 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sadra-barikbin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vfdev-5 commented Sep 18, 2023

Uh oh!

Uh oh!

Uh oh!

sihyeong671 commented Sep 18, 2023

Uh oh!

vfdev-5 commented Sep 18, 2023

Uh oh!

sihyeong671 commented Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vfdev-5 commented Sep 18, 2023

Uh oh!

Uh oh!

Uh oh!

vfdev-5 commented Sep 19, 2023

Uh oh!

Uh oh!

Uh oh!

vfdev-5 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sihyeong671 commented Sep 16, 2023 •

edited by vfdev-5

Loading

sihyeong671 commented Sep 18, 2023 •

edited

Loading