refactor intialize bucket views by zhaojuanmao · Pull Request #44330 · pytorch/pytorch

zhaojuanmao · 2020-09-08T20:47:57Z

Stack from ghstack:

[reland] move rebuild buckets from end of first iteration to beginning of second iteration #44798 [reland] move rebuild buckets from end of first iteration to beginning of second iteration
refactor intialize bucket views #44330 refactor intialize bucket views

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well

Differential Revision: D23583347

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) ghstack-source-id: 111612373 Pull Request resolved: #44330

dr-ci · 2020-09-08T21:31:15Z

💊 CI failures summary and remediations

As of commit f4cf604 (more details on the Dr. CI page):

1/2 failures introduced in this PR
1/2 broken upstream at merge base 6befc09 on Sep 16 from 3:30pm to 5:20pm PDT (8 commits; 7b3432c - 3fa7f51)

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_legacy_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 17 04:47:35 ERROR [5.765s]: test_tensorexpr_cpp_cuda (test_jit.TestJit)

Sep 17 04:47:35   test_factory_ops_requires_grad_fail (jit.test_unsupported_ops.TestUnsupportedOps) ... ok (0.004s) 
Sep 17 04:47:35   test_init_ops (jit.test_unsupported_ops.TestUnsupportedOps) ... ok (0.021s) 
Sep 17 04:47:35   test_ops_bound_in_functional (jit.test_unsupported_ops.TestUnsupportedOps) ... ok (0.003s) 
Sep 17 04:47:35   test_with_as (jit.test_with.TestWith) ... ok (0.083s) 
Sep 17 04:47:35   test_with_errors (jit.test_with.TestWith) ... ok (0.016s) 
Sep 17 04:47:35   test_with_exceptions (jit.test_with.TestWith) ... ok (0.007s) 
Sep 17 04:47:35   test_with_no_as (jit.test_with.TestWith) ... ok (0.086s) 
Sep 17 04:47:35   test_with_no_grad (jit.test_with.TestWith) ... ok (0.021s) 
Sep 17 04:47:35  
Sep 17 04:47:35 ====================================================================== 
Sep 17 04:47:35 ERROR [5.765s]: test_tensorexpr_cpp_cuda (test_jit.TestJit) 
Sep 17 04:47:35 ---------------------------------------------------------------------- 
Sep 17 04:47:35 Traceback (most recent call last): 
Sep 17 04:47:35   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 818, in wrapper 
Sep 17 04:47:35     method(*args, **kwargs) 
Sep 17 04:47:35   File "/var/lib/jenkins/workspace/test/test_jit.py", line 1431, in test_tensorexpr_cpp_cuda 
Sep 17 04:47:35     torch._C._run_tensorexpr_cpp_tests(run_cuda=True) 
Sep 17 04:47:35 RuntimeError: std::fabs((v1[kPaddingSize + i]) - (v2[kPaddingSize + i])) < (abs_error) INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/test/cpp/tensorexpr/padded_buffer.h":258, please report a bug to PyTorch. index: 0, v1: (b_v, 323584), v2: (b_ref, 778240) 
Sep 17 04:47:35  
Sep 17 04:47:35 ---------------------------------------------------------------------- 
Sep 17 04:47:35 Ran 2716 tests in 271.342s

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

Since your merge base is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Check out the recency history of this "viable master" tracking branch.

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test on Sep 16 from 3:30pm to 5:20pm PDT (8 commits; 7b3432c - 3fa7f51)
- 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 38 times.

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 111631226 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 111792886 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

codecov · 2020-09-10T22:16:08Z

Codecov Report

❗ No coverage uploaded for pull request base (gh/zhaojuanmao/52/base@7c3371f). Click here to learn what that means.
The diff coverage is n/a.

@@                    Coverage Diff                    @@
##             gh/zhaojuanmao/52/base   #44330   +/-   ##
=========================================================
  Coverage                          ?   68.00%           
=========================================================
  Files                             ?      384           
  Lines                             ?    49602           
  Branches                          ?        0           
=========================================================
  Hits                              ?    33731           
  Misses                            ?    15871           
  Partials                          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c3371f...2498a46. Read the comment docs.

torch/csrc/distributed/c10d/reducer.h

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 112022404 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

[test all] Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 112185672 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

[test all] Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 112243783 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/) [ghstack-poisoned]

[test all] Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 112257271 Differential Revision: [D23583347](https://our.internmc.facebook.com/intern/diff/D23583347/)

zhaojuanmao · 2020-09-17T14:02:31Z

tests in this PR and ci-all tests in #44865 passed, failed tests are not related, landing

facebook-github-bot · 2020-09-17T18:14:48Z

This pull request has been merged in d2b4534.

Summary: [test all] Pull Request resolved: #44330 Part of relanding PR #41954, this refactor is to seperate intialize_bucket_views and populate_bucket_views_out, as they are doing different things and called by different callsites as well ghstack-source-id: 112257271 Test Plan: unit tests Reviewed By: mrshenli Differential Revision: D23583347 fbshipit-source-id: a5f2041b2c4f2c2b5faba1af834c7143eaade938

zhaojuanmao requested review from mrshenli and pietern as code owners September 8, 2020 20:47

zhaojuanmao mentioned this pull request Sep 8, 2020

move rebuild buckets from end of first iteration to beginning of second iteration #44326

Closed

zhaojuanmao mentioned this pull request Sep 8, 2020

[reland] Make grad point to bucket buffer in DDP to save memory usage #44344

Closed

mrshenli approved these changes Sep 14, 2020

View reviewed changes

torch/csrc/distributed/c10d/reducer.h Outdated Show resolved Hide resolved

torch/csrc/distributed/c10d/reducer.h Outdated Show resolved Hide resolved

torch/csrc/distributed/c10d/reducer.h Outdated Show resolved Hide resolved

This was referenced Sep 16, 2020

[reland] move rebuild buckets from end of first iteration to beginning of second iteration #44798

Closed

[For ci-all tests only] refactor intialize bucket views #44838

Closed

zhaojuanmao requested review from pritamdamania87 and rohan-varma as code owners September 16, 2020 23:29

zhaojuanmao mentioned this pull request Sep 17, 2020

[For ci-all tests] refactor intialize bucket views #44865

Closed

facebook-github-bot closed this in d2b4534 Sep 17, 2020

facebook-github-bot added the merged label Sep 17, 2020

facebook-github-bot deleted the gh/zhaojuanmao/52/head branch September 21, 2020 14:14

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor intialize bucket views#44330

refactor intialize bucket views#44330
zhaojuanmao wants to merge 8 commits intogh/zhaojuanmao/52/basefrom
gh/zhaojuanmao/52/head

zhaojuanmao commented Sep 8, 2020 •

edited

Loading

Uh oh!

dr-ci bot commented Sep 8, 2020 •

edited

Loading

Uh oh!

codecov bot commented Sep 10, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhaojuanmao commented Sep 17, 2020

Uh oh!

facebook-github-bot commented Sep 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zhaojuanmao commented Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_legacy_test (1/1)

🚧 1 fixed upstream failure:

Uh oh!

codecov bot commented Sep 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhaojuanmao commented Sep 17, 2020

Uh oh!

facebook-github-bot commented Sep 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhaojuanmao commented Sep 8, 2020 •

edited

Loading

dr-ci bot commented Sep 8, 2020 •

edited

Loading

codecov bot commented Sep 10, 2020 •

edited

Loading