Complex gradcheck logic #43208

anjali411 · 2020-08-18T14:23:55Z

Stack from ghstack:

[WIP] Add tests for R to C cases #44744 Add tests for R to C cases
Update backward definition for more operators and reenable tests in test_ops.py #44444 [WIP] Update backward definition for more operators and reenable tests in test_ops.py
Complex gradcheck logic #43208 Complex gradcheck logic

This PR adds gradcheck for complex. The logic used for complex gradcheck is described in Section 3.5.3 here: https://arxiv.org/pdf/1701.00392.pdf

More concretely, this PR introduces the following changes:

Updates get_numerical_jacobian to take as input a scalar value for vector (v). Adds gradcheck logic for C -> C, C-> R, R -> C. For R -> C functions, only the real value of gradient is propagated.
Adds backward definition for torch.complex and also adds a test to verify the definition added.
Updates backward for mul, sin, cos, sinh, cosh.
Adds tests for all torch.real, torch.imag, torch.view_as_real, torch.view_as_complex, torch.conj.

Follow up tasks:

Add more thorough tests for R -> C cases. Specifically, add R->C test variants for functions. for e.g., torch.mul(complex_tensor, real_tensor) ([WIP] Add tests for R to C cases #44744 )
Add back commented test in common_methods_invocation.py. ( Update backward definition for more operators and reenable tests in test_ops.py #44444 )
Add more special case checking for complex gradcheck to make debugging easier.
Update complex autograd note.
disable complex autograd for operators not tested for complex. (Add allowlist for complex backward #45461 )
Re-enable tests in test_ops.py for complex dtype. ( Update backward definition for more operators and reenable tests in test_ops.py #44444 )
Re-enable TestGradCheckOverride.test_gradcheck cc. @hameerabbasi (Allow Tensor-likes in torch.autograd.gradcheck #45732 )

Differential Revision: D23655088

[ghstack-poisoned]

dr-ci · 2020-08-18T14:32:03Z

💊 CI failures summary and remediations

As of commit 87ed330 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

1 job timed out:

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_test

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 265 times.

[ghstack-poisoned]

ghstack-source-id: 6a2e586 Pull Request resolved: #43208

[ghstack-poisoned]

ghstack-source-id: 10a83f3 Pull Request resolved: #43208

[ghstack-poisoned]

ghstack-source-id: 62bde11 Pull Request resolved: #43208

[ghstack-poisoned]

ghstack-source-id: 2b25ff1 Pull Request resolved: #43208

[ghstack-poisoned]

To unblock #43208, which adds "is_complex" checks to backward formulas that are being tested for batched gradient support with vmap. Test Plan: - `pytest test/test_vmap.py -v` ghstack-source-id: f6dcd35 Pull Request resolved: #44649

To unblock #43208, which adds "is_complex" checks to backward formulas that are being tested for batched gradient support with vmap. Test Plan: - `pytest test/test_vmap.py -v` Differential Revision: [D23685356](https://our.internmc.facebook.com/intern/diff/D23685356) [ghstack-poisoned]

To unblock #43208, which adds "is_complex" checks to backward formulas that are being tested for batched gradient support with vmap. Test Plan: - `pytest test/test_vmap.py -v` ghstack-source-id: 596cb16 Pull Request resolved: #44649

@hameerabbasi

This PR adds gradcheck for complex. The logic used for complex gradcheck is described in Section 3.5.3 here: https://arxiv.org/pdf/1701.00392.pdf More concretely, this PR introduces the following changes: 1. Updates get_numerical_jacobian to take as input a scalar value for vector (v). Adds gradcheck logic for C -> C, C-> R, R -> C. For R -> C functions, only the real value of gradient is propagated. 2. Adds backward definition for `torch.complex` and also adds a test to verify the definition added. 3. Updates backward for `mul`, `sin`, `cos`, `sinh`, `cosh`. 4. Adds tests for all `torch.real`, `torch.imag`, `torch.view_as_real`, `torch.view_as_complex`, `torch.conj`. Follow up tasks: - [ ] Add more thorough tests for R -> C cases. Specifically, add R->C test variants for functions. for e.g., `torch.mul(complex_tensor, real_tensor)` (#44744 ) - [ ] Add back commented test in `common_methods_invocation.py`. ( #43208 ) - [ ] Add more special case checking for complex gradcheck to make debugging easier. - [ ] Update complex autograd note. - [ ] disable complex autograd for operators not tested for complex. - [ ] Re-enable tests in `test_ops.py` for complex dtype. ( #43208 ) - [ ] Re-enable `TestGradCheckOverride.test_gradcheck` cc. @hameerabbasi Differential Revision: [D23655088](https://our.internmc.facebook.com/intern/diff/D23655088) [ghstack-poisoned]

@hameerabbasi

This PR adds gradcheck for complex. The logic used for complex gradcheck is described in Section 3.5.3 here: https://arxiv.org/pdf/1701.00392.pdf More concretely, this PR introduces the following changes: 1. Updates get_numerical_jacobian to take as input a scalar value for vector (v). Adds gradcheck logic for C -> C, C-> R, R -> C. For R -> C functions, only the real value of gradient is propagated. 2. Adds backward definition for `torch.complex` and also adds a test to verify the definition added. 3. Updates backward for `mul`, `sin`, `cos`, `sinh`, `cosh`. 4. Adds tests for all `torch.real`, `torch.imag`, `torch.view_as_real`, `torch.view_as_complex`, `torch.conj`. Follow up tasks: - [ ] Add more thorough tests for R -> C cases. Specifically, add R->C test variants for functions. for e.g., `torch.mul(complex_tensor, real_tensor)` (#44744 ) - [ ] Add back commented test in `common_methods_invocation.py`. ( #43208 ) - [ ] Add more special case checking for complex gradcheck to make debugging easier. - [ ] Update complex autograd note. - [ ] disable complex autograd for operators not tested for complex. - [ ] Re-enable tests in `test_ops.py` for complex dtype. ( #43208 ) - [ ] Re-enable `TestGradCheckOverride.test_gradcheck` cc. @hameerabbasi Differential Revision: [D23655088](https://our.internmc.facebook.com/intern/diff/D23655088) [ghstack-poisoned]

anjali411 · 2020-09-16T15:54:37Z

torch/autograd/gradcheck.py

+        # or pure imaginary delta
+        def compute_gradient(delta):
+            # we currently assume that the norm of delta equals eps
+            assert(delta == eps or delta == (eps * 1j))


cc. @ngimel

Summary: Pull Request resolved: #44649 To unblock #43208, which adds "is_complex" checks to backward formulas that are being tested for batched gradient support with vmap. Test Plan: - `pytest test/test_vmap.py -v` Reviewed By: anjali411 Differential Revision: D23685356 Pulled By: zou3519 fbshipit-source-id: 29e41a9296336f6d1008e3040cade4c643bf5ebf

@hameerabbasi

This PR adds gradcheck for complex. The logic used for complex gradcheck is described in Section 3.5.3 here: https://arxiv.org/pdf/1701.00392.pdf More concretely, this PR introduces the following changes: 1. Updates get_numerical_jacobian to take as input a scalar value for vector (v). Adds gradcheck logic for C -> C, C-> R, R -> C. For R -> C functions, only the real value of gradient is propagated. 2. Adds backward definition for `torch.complex` and also adds a test to verify the definition added. 3. Updates backward for `mul`, `sin`, `cos`, `sinh`, `cosh`. 4. Adds tests for all `torch.real`, `torch.imag`, `torch.view_as_real`, `torch.view_as_complex`, `torch.conj`. Follow up tasks: - [ ] Add more thorough tests for R -> C cases. Specifically, add R->C test variants for functions. for e.g., `torch.mul(complex_tensor, real_tensor)` (#44744 ) - [ ] Add back commented test in `common_methods_invocation.py`. ( #43208 ) - [ ] Add more special case checking for complex gradcheck to make debugging easier. - [ ] Update complex autograd note. - [ ] disable complex autograd for operators not tested for complex. - [ ] Re-enable tests in `test_ops.py` for complex dtype. ( #43208 ) - [ ] Re-enable `TestGradCheckOverride.test_gradcheck` cc. @hameerabbasi Differential Revision: [D23655088](https://our.internmc.facebook.com/intern/diff/D23655088) [ghstack-poisoned]

@hameerabbasi

This PR adds gradcheck for complex. The logic used for complex gradcheck is described in Section 3.5.3 here: https://arxiv.org/pdf/1701.00392.pdf More concretely, this PR introduces the following changes: 1. Updates get_numerical_jacobian to take as input a scalar value for vector (v). Adds gradcheck logic for C -> C, C-> R, R -> C. For R -> C functions, only the real value of gradient is propagated. 2. Adds backward definition for `torch.complex` and also adds a test to verify the definition added. 3. Updates backward for `mul`, `sin`, `cos`, `sinh`, `cosh`. 4. Adds tests for all `torch.real`, `torch.imag`, `torch.view_as_real`, `torch.view_as_complex`, `torch.conj`. Follow up tasks: - [ ] Add more thorough tests for R -> C cases. Specifically, add R->C test variants for functions. for e.g., `torch.mul(complex_tensor, real_tensor)` (#44744 ) - [ ] Add back commented test in `common_methods_invocation.py`. ( #43208 ) - [ ] Add more special case checking for complex gradcheck to make debugging easier. - [ ] Update complex autograd note. - [ ] disable complex autograd for operators not tested for complex. - [ ] Re-enable tests in `test_ops.py` for complex dtype. ( #43208 ) - [ ] Re-enable `TestGradCheckOverride.test_gradcheck` cc. @hameerabbasi Differential Revision: [D23655088](https://our.internmc.facebook.com/intern/diff/D23655088) [ghstack-poisoned]

Summary: Pull Request resolved: #44649 To unblock #43208, which adds "is_complex" checks to backward formulas that are being tested for batched gradient support with vmap. Test Plan: - `pytest test/test_vmap.py -v` Reviewed By: anjali411 Differential Revision: D23685356 Pulled By: zou3519 fbshipit-source-id: 29e41a9296336f6d1008e3040cade4c643bf5ebf

facebook-github-bot · 2020-09-21T06:15:12Z

@anjali411 merged this pull request in 9f67176.

anjali411 · 2020-09-22T15:50:40Z

@hameerabbasi this PR has been merged so would you like to create a follow-up PR to re-enable TestGradCheckOverride.test_gradcheck?

hameerabbasi · 2020-09-22T15:51:46Z

I'll work on that soon, thanks.

ghstack-source-id: fdc0e91 Pull Request resolved: pytorch/pytorch#43208

anjali411 · 2020-10-01T15:29:54Z

I'll work on that soon, thanks.

Hi @hameerabbasi just checking in if you are still planning to work on re-enabling TestGradCheckOverride.test_gradcheck

hameerabbasi · 2020-10-01T15:33:15Z

I apologize @anjali411, that fell off the radar due to a personal emergency. I'll add it to my to-do list so it's no longer just in my head.

anjali411 · 2020-10-01T15:41:22Z

I apologize @anjali411, that fell off the radar due to a personal emergency. I'll add it to my to-do list so it's no longer just in my head.

no worries and hope all is well! feel free to add me as a reviewer once you have a PR ready with your changes.

anjali411 · 2020-10-09T03:04:38Z

@hameerabbasi gentle ping for re-enabling TestGradCheckOverride.test_gradcheck

hameerabbasi · 2020-10-09T05:36:36Z

There's a PR up that I submitted yesterday, #45732. Requested your review on it.

anjali411 · 2020-10-09T13:31:25Z

There's a PR up that I submitted yesterday, #45732. Requested your review on it.

thanks @hameerabbasi it looks great. I totally missed your PR before, but looks like @albanD is already reviewing it :D

Complex autograd logic

c98cea7

[ghstack-poisoned]

anjali411 requested review from albanD and apaszke as code owners August 18, 2020 14:23

anjali411 mentioned this pull request Aug 18, 2020

Autograd gradcheck code clean up #43167

Closed

anjali411 changed the title ~~Complex autograd logic~~ [WIP] Complex autograd logic Aug 18, 2020

anjali411 changed the title ~~[WIP] Complex autograd logic~~ [WIP] Complex gradcheck logic Aug 18, 2020

Update on "[WIP] Complex gradcheck logic"

4115deb

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

2ac87cf

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Aug 18, 2020

Complex autograd logic

3a08a16

ghstack-source-id: 6a2e586 Pull Request resolved: #43208

Update on "[WIP] Complex gradcheck logic"

9c555c4

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

d3f2042

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Aug 19, 2020

Complex autograd logic

69137e4

ghstack-source-id: 10a83f3 Pull Request resolved: #43208

Update on "[WIP] Complex gradcheck logic"

a4c924b

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

ee11d18

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

40346a4

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Aug 25, 2020

Complex autograd logic

576b09c

ghstack-source-id: 62bde11 Pull Request resolved: #43208

Update on "[WIP] Complex gradcheck logic"

7a09843

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

999bc75

[ghstack-poisoned]

Update on "[WIP] Complex gradcheck logic"

4cfd4eb

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Aug 26, 2020

Complex autograd logic

865aebf

ghstack-source-id: 2b25ff1 Pull Request resolved: #43208

Update on "[WIP] Complex gradcheck logic"

63e3451

[ghstack-poisoned]

anjali411 mentioned this pull request Sep 15, 2020

[WIP] Add tests for R to C cases #44744

Closed

anjali411 commented Sep 16, 2020

View reviewed changes

facebook-github-bot closed this in 9f67176 Sep 21, 2020

facebook-github-bot added the merged label Sep 21, 2020

IvanYashchuk mentioned this pull request Sep 21, 2020

Autograd not working for torch.exp(1j * phase) #43349

Closed

loadbxh pushed a commit to loadbxh/Torch that referenced this pull request Sep 23, 2020

Complex autograd logic

4d15f14

ghstack-source-id: fdc0e91 Pull Request resolved: pytorch/pytorch#43208

facebook-github-bot deleted the gh/anjali411/52/head branch September 24, 2020 14:21

anjali411 mentioned this pull request Oct 1, 2020

Complex Numbers Support #33152

Closed

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Complex gradcheck logic #43208

Complex gradcheck logic #43208

Uh oh!

anjali411 commented Aug 18, 2020 •

edited

Loading

Uh oh!

dr-ci bot commented Aug 18, 2020 •

edited

Loading

Uh oh!

anjali411 Sep 16, 2020

Uh oh!

facebook-github-bot commented Sep 21, 2020

Uh oh!

anjali411 commented Sep 22, 2020

Uh oh!

hameerabbasi commented Sep 22, 2020

Uh oh!

anjali411 commented Oct 1, 2020

Uh oh!

hameerabbasi commented Oct 1, 2020

Uh oh!

anjali411 commented Oct 1, 2020

Uh oh!

anjali411 commented Oct 9, 2020 •

edited

Loading

Uh oh!

hameerabbasi commented Oct 9, 2020

Uh oh!

anjali411 commented Oct 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Complex gradcheck logic #43208

Complex gradcheck logic #43208

Uh oh!

Conversation

anjali411 commented Aug 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Aug 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

anjali411 Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 21, 2020

Uh oh!

anjali411 commented Sep 22, 2020

Uh oh!

hameerabbasi commented Sep 22, 2020

Uh oh!

anjali411 commented Oct 1, 2020

Uh oh!

hameerabbasi commented Oct 1, 2020

Uh oh!

anjali411 commented Oct 1, 2020

Uh oh!

anjali411 commented Oct 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hameerabbasi commented Oct 9, 2020

Uh oh!

anjali411 commented Oct 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

anjali411 commented Aug 18, 2020 •

edited

Loading

dr-ci bot commented Aug 18, 2020 •

edited

Loading

anjali411 commented Oct 9, 2020 •

edited

Loading