Skip to content

Implement python API for AdaBelief optimizer#50157

Closed
juntang-zhuang wants to merge 4 commits intopytorch:masterfrom
juntang-zhuang:adabelief_python
Closed

Implement python API for AdaBelief optimizer#50157
juntang-zhuang wants to merge 4 commits intopytorch:masterfrom
juntang-zhuang:adabelief_python

Conversation

@juntang-zhuang
Copy link
Copy Markdown

@juntang-zhuang juntang-zhuang commented Jan 6, 2021

Fixes #47571 partially, add python API for AdaBelief optimizer.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jan 6, 2021 •
edited
Loading

💊 CI failures summary and remediations

As of commit 92938c6 (more details on the Dr. CI page):

  • 8/8 failures possibly* introduced in this PR
    • 1/8 non-CircleCI failure(s)

🕵️ 7 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (1/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:09:03 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]") [assignment] 
Jan 06 22:08:42   test_type_hint_examples (__main__.TestTypeHints)
Jan 06 22:09:03 Runs mypy over all the test examples present in ... ok (20.698s)
Jan 06 22:09:03 
Jan 06 22:09:03 ======================================================================
Jan 06 22:09:03 FAIL [77.961s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:09:03 Runs mypy over all files specified in mypy.ini
Jan 06 22:09:03 ----------------------------------------------------------------------
Jan 06 22:09:03 Traceback (most recent call last):
Jan 06 22:09:03   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:09:03     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:09:03 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:09:03 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:09:03 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:09:03 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:09:03  
Jan 06 22:09:03 
Jan 06 22:09:03 ----------------------------------------------------------------------
Jan 06 22:09:03 Ran 4 tests in 115.840s
Jan 06 22:09:03 
Jan 06 22:09:03 FAILED (failures=1)
Jan 06 22:09:03

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (2/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:19:50 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]") [assignment] 
Jan 06 22:19:29   test_run_mypy (__main__.TestTypeHints) ... FAIL (64.640s)
Jan 06 22:19:31   test_run_mypy_strict (__main__.TestTypeHints) ... ok (2.777s)
Jan 06 22:19:50   test_type_hint_examples (__main__.TestTypeHints) ... ok (18.458s)
Jan 06 22:19:50 
Jan 06 22:19:50 ======================================================================
Jan 06 22:19:50 FAIL [64.640s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:19:50 ----------------------------------------------------------------------
Jan 06 22:19:50 Traceback (most recent call last):
Jan 06 22:19:50   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:19:50     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:19:50 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:19:50 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:19:50 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:19:50 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:19:50  
Jan 06 22:19:50 
Jan 06 22:19:50 ----------------------------------------------------------------------
Jan 06 22:19:50 Ran 4 tests in 97.674s
Jan 06 22:19:50 
Jan 06 22:19:50 FAILED (failures=1)
Jan 06 22:19:50

See CircleCI build pytorch_doc_test (3/7)

Step: "Doc test" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:02:18 sccache: error: couldn't connect to server 
Jan 06 22:02:18 ++++ eval 'extract_trap_cmd '
Jan 06 22:02:18 +++++ extract_trap_cmd
Jan 06 22:02:18 +++++ printf '%s\n' ''
Jan 06 22:02:18 ++++ printf '%s\n' cleanup
Jan 06 22:02:18 +++ trap -- '
Jan 06 22:02:18 cleanup' EXIT
Jan 06 22:02:18 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build != *pytorch-win-* ]]
Jan 06 22:02:18 +++ which sccache
Jan 06 22:02:18 +++ sccache --stop-server
Jan 06 22:02:18 Stopping sccache server...
Jan 06 22:02:18 sccache: error: couldn't connect to server
Jan 06 22:02:18 sccache: caused by: Connection refused (os error 111)
Jan 06 22:02:18 +++ true
Jan 06 22:02:18 +++ rm /var/lib/jenkins/sccache_error.log
Jan 06 22:02:18 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build == *rocm* ]]
Jan 06 22:02:18 +++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
Jan 06 22:02:18 +++ SCCACHE_IDLE_TIMEOUT=1200
Jan 06 22:02:18 +++ RUST_LOG=sccache::server=error
Jan 06 22:02:18 +++ sccache --start-server
Jan 06 22:02:18 sccache: Starting the server...
Jan 06 22:02:19 +++ sccache --zero-stats

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_test (4/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:06:15 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]") [assignment] 
Jan 06 22:05:49   test_run_mypy (__main__.TestTypeHints) ... FAIL (68.339s)
Jan 06 22:05:52   test_run_mypy_strict (__main__.TestTypeHints) ... ok (3.434s)
Jan 06 22:06:15   test_type_hint_examples (__main__.TestTypeHints) ... ok (22.495s)
Jan 06 22:06:15 
Jan 06 22:06:15 ======================================================================
Jan 06 22:06:15 FAIL [68.339s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:06:15 ----------------------------------------------------------------------
Jan 06 22:06:15 Traceback (most recent call last):
Jan 06 22:06:15   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:06:15     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:06:15 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:06:15 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:06:15 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:06:15 Found 3 errors in 1 file (checked 1189 source files)
Jan 06 22:06:15  
Jan 06 22:06:15 
Jan 06 22:06:15 ----------------------------------------------------------------------
Jan 06 22:06:15 Ran 4 tests in 106.299s
Jan 06 22:06:15 
Jan 06 22:06:15 FAILED (failures=1)
Jan 06 22:06:15

See CircleCI build pytorch_python_doc_build (5/7)

Step: "Doc Build and Push" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:15:12 Makefile:38: recipe for target 'html' failed 
Jan 06 22:15:11 copying images... [ 96%] _static/img/tensorboard/add_hparam.png
Jan 06 22:15:11 copying images... [100%] scripts/activation_images/ReLU6.png
Jan 06 22:15:11 
Jan 06 22:15:12 copying static files... ... done
Jan 06 22:15:12 copying extra files... done
Jan 06 22:15:12 dumping search index in English (code: en)... done
Jan 06 22:15:12 dumping object inventory... done
Jan 06 22:15:12 build finished with problems, 4 warnings.
Jan 06 22:15:12 /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme/pytorch_sphinx_theme/search.html:21: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
Jan 06 22:15:12   {% trans %}Please activate JavaScript to enable the search
Jan 06 22:15:12 Makefile:38: recipe for target 'html' failed
Jan 06 22:15:12 make: *** [html] Error 1


Exited with code exit status 2

See CircleCI build pytorch_linux_xenial_py3_clang5_asan_test1 (6/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:04:54 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]") [assignment] 
Jan 06 22:04:28   test_run_mypy (__main__.TestTypeHints) ... FAIL (69.953s)
Jan 06 22:04:31   test_run_mypy_strict (__main__.TestTypeHints) ... ok (3.298s)
Jan 06 22:04:54   test_type_hint_examples (__main__.TestTypeHints) ... ok (22.656s)
Jan 06 22:04:54 
Jan 06 22:04:54 ======================================================================
Jan 06 22:04:54 FAIL [69.953s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:04:54 ----------------------------------------------------------------------
Jan 06 22:04:54 Traceback (most recent call last):
Jan 06 22:04:54   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:04:54     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:04:54 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:04:54 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:04:54 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:04:54 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:04:54  
Jan 06 22:04:54 
Jan 06 22:04:54 ----------------------------------------------------------------------
Jan 06 22:04:54 Ran 4 tests in 108.797s
Jan 06 22:04:54 
Jan 06 22:04:54 FAILED (failures=1)
Jan 06 22:04:54

See CircleCI build pytorch_linux_bionic_py3_6_clang9_test (7/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 06 22:02:47 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]") [assignment] 
Jan 06 22:02:25   test_run_mypy (__main__.TestTypeHints) ... FAIL (60.092s)
Jan 06 22:02:28   test_run_mypy_strict (__main__.TestTypeHints) ... ok (2.858s)
Jan 06 22:02:47   test_type_hint_examples (__main__.TestTypeHints) ... ok (19.526s)
Jan 06 22:02:47 
Jan 06 22:02:47 ======================================================================
Jan 06 22:02:47 FAIL [60.092s]: test_run_mypy (__main__.TestTypeHints)
Jan 06 22:02:47 ----------------------------------------------------------------------
Jan 06 22:02:47 Traceback (most recent call last):
Jan 06 22:02:47   File "test_type_hints.py", line 214, in test_run_mypy
Jan 06 22:02:47     self.fail(f"mypy failed: {stdout} {stderr}")
Jan 06 22:02:47 AssertionError: mypy failed: torch/optim/_multi_tensor/adabelief.py:162: error: Incompatible types in assignment (expression has type "Union[Tuple[Tensor, ...], List[Tensor]]", variable has type "List[Any]")  [assignment]
Jan 06 22:02:47 torch/optim/_multi_tensor/adabelief.py:250: error: Argument 1 of "zero_grad" is incompatible with supertype "Optimizer"; supertype defines the argument type as "Optional[bool]"  [override]
Jan 06 22:02:47 torch/optim/_multi_tensor/adabelief.py:251: error: Need type annotation for 'per_device_and_dtype_grads'  [var-annotated]
Jan 06 22:02:47 Found 3 errors in 1 file (checked 1185 source files)
Jan 06 22:02:47  
Jan 06 22:02:47 
Jan 06 22:02:47 ----------------------------------------------------------------------
Jan 06 22:02:47 Ran 4 tests in 93.505s
Jan 06 22:02:47 
Jan 06 22:02:47 FAILED (failures=1)
Jan 06 22:02:47

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@glaringlee
Copy link
Copy Markdown
Contributor

glaringlee commented Jan 6, 2021 •
edited
Loading

@juntang-zhuang
Hey Juntang, please see the comment from my colleague who is working on python side of pytorch. It seems we have an adoption rule for new features on python side which I am not aware of. Sry about this, but I think this is a good feature as the paper is already there, more and more people will use it and we probably can adopt this into pytorch main branch later I think.

here is the link: #47571

@juntang-zhuang
Copy link
Copy Markdown
Author

juntang-zhuang commented Jan 6, 2021

@glaringlee Thanks a lot for the update. I'll just keep the changes in my local repository so we can consider merging it later.

@glaringlee
Copy link
Copy Markdown
Contributor

glaringlee commented Jan 6, 2021 •
edited
Loading

@glaringlee Thanks a lot for the update. I'll just keep the changes in my local repository so we can consider merging it later.

Sure, let's revisit this in the middle of the year see whether you have more citations, or if you optimizer is used by a large facility (eg, some big companies?), I think that would be a big plus. let's see.

@H-Huang H-Huang added module: optimizer Related to torch.optim triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jan 6, 2021
@spandantiwari spandantiwari mentioned this pull request Jan 12, 2021
Closed
@iramazanli iramazanli self-requested a review August 30, 2021 15:27
iramazanli
iramazanli reviewed Aug 30, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@iramazanli iramazanli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for proposing AdaBelief implementation to PyTorch Core Library.

We would want to refer to rulebook for adding optimizers to PyTorch Core:
https://github.com/pytorch/pytorch/wiki/Developer-FAQ#i-have-implemented-an-algorithm--feature-from-a-research-paper-that-is-related-to-optimization-when-is-the-correct-time-to-add-it-to-pytorch-core-

Which implies although AdaBelief is a very promising algorithm, given that it's relatively new algorithm we cannot add it to Core Library yet.

It can be interesting discussion to return back to this issue in the following years.

@iramazanli iramazanli closed this Aug 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@iramazanli iramazanli iramazanli left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed module: optimizer Related to torch.optim open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Adabelief optimizer to the C++ API

6 participants

@juntang-zhuang @facebook-github-bot @glaringlee @iramazanli @H-Huang @pytorchbot