Split test_c10d.py to test_c10d_common.py, test_c10d_gloo.py, test_c10d_nccl.py by pbelevich · Pull Request #56598 · pytorch/pytorch

pbelevich · 2021-04-21T17:56:31Z

Test Plan: NA

Differential Revision: D27913170

facebook-github-bot · 2021-04-21T17:56:38Z

💊 CI failures summary and remediations

As of commit 751bf5e (more details on the Dr. CI page):

5/7 failures possibly* introduced in this PR
- 1/5 non-scanned failure(s)
2/7 broken upstream at merge base 76fbd75 since Apr 21

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_windows_vs2019_py36_cuda10.1_test1 (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: Found no NVIDIA driver on your sy...ver from http://www.nvidia.com/Download/index.aspx

1554:16cc @ 00550265 - LdrpGetProcedureAddress - INFO: Locating procedure "FlsGetValue" by name
(1554.16cc): C++ EH exception - code e06d7363 (first chance)
(1554.16cc): C++ EH exception - code e06d7363 (first chance)
(1554.16cc): C++ EH exception - code e06d7363 (first chance)
(1554.16cc): C++ EH exception - code e06d7363 (first chance)
(1554.16cc): C++ EH exception - code e06d7363 (first chance)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\circleci\project\build\win_tmp\build\torch\cuda\__init__.py", line 170, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
1554:16cc @ 00550312 - LdrLoadDll - ENTER: DLL name: api-ms-win-appmodel-runtime-l1-1-2
1554:16cc @ 00550312 - LdrpPreprocessDllName - INFO: DLL api-ms-win-appmodel-runtime-l1-1-2 was redirected to C:\Windows\SYSTEM32\kernel.appcore.dll by API set
1554:16cc @ 00550312 - LdrpLoadDllInternal - ENTER: DLL name: C:\Windows\SYSTEM32\kernel.appcore.dll
1554:16cc @ 00550328 - LdrpFindKnownDll - ENTER: DLL name: kernel.appcore.dll
1554:16cc @ 00550328 - LdrpFindKnownDll - RETURN: Status: 0x00000000
1554:16cc @ 00550328 - LdrpMinimalMapModule - ENTER: DLL name: C:\Windows\System32\kernel.appcore.dll
ModLoad: 00007ffb`fd1a0000 00007ffb`fd1b1000   C:\Windows\System32\kernel.appcore.dll
1554:16cc @ 00550328 - LdrpMinimalMapModule - RETURN: Status: 0x00000000
1554:16cc @ 00550328 - LdrpFindDllActivationContext - INFO: Probing for the manifest of DLL "C:\Windows\System32\kernel.appcore.dll" failed with status 0xc000008a
1554:16cc @ 00550328 - LdrpPreprocessDllName - INFO: DLL api-ms-win-core-profile-l1-1-0.dll was redirected to C:\Windows\SYSTEM32\kernelbase.dll by API set

3 failures not recognized by patterns:

Job	Step	Action
^{pytorch_linux_xenial_py3_clang5_asan_test2}	^{Run tests}	🔁 rerun
^{pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2}	^{Run tests}	🔁 rerun
^{pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1}	^{Run tests}	🔁 rerun

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet.

pytorch_linux_xenial_py3_clang7_onnx_ort_test1 since Apr 21 (24ff92f)
- 🔁 rerun

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

pytorch_linux_xenial_py3_clang7_onnx_ort_test2 on Apr 21 from 5:59pm to 7:44pm (24ff92f - 6032ea0)
- 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

facebook-github-bot · 2021-04-21T17:56:47Z

This pull request was exported from Phabricator. Differential Revision: D27913170

facebook-github-bot · 2021-04-21T19:39:44Z

This pull request was exported from Phabricator. Differential Revision: D27913170

facebook-github-bot · 2021-04-21T20:05:41Z

This pull request was exported from Phabricator. Differential Revision: D27913170

wayi1 · 2021-04-21T22:25:49Z

Thanks for the refactoring! This will definitely make the test codebase much more manageable.

On the other hand, I think this PR does a bit more than plain splitting. Therefore, it will be better to add some descriptions in the PR to highlight some changes, e.g., creating some abstract classes like AbstractCommTest, AbstractDistributedDataParallelTest and AbstractTimeoutTest.

Another suggestion is separating out many fundamental store-related unit tests into a separate file, as these tests and other distributed training tests target different layers. This can be done in a fllow-up PR if you'd like.

wayi1 · 2021-04-21T22:26:29Z

test/distributed/test_c10d_gloo.py

Why is this test in gloo rather than common?

line 181:

c10d.init_process_group(backend="gloo", init_method="env://")

Better to add @requires_gloo() here.

wayi1 · 2021-04-21T22:30:28Z

test/distributed/test_c10d_gloo.py

The indentation here and a few "ValueError" lines and "RuntimeError" lines are changed. Previously it's aligned with the end of "with". Is this intentional?

wayi1 · 2021-04-21T22:32:25Z

test/distributed/test_c10d_gloo.py

Indentation changed.

facebook-github-bot · 2021-04-21T23:41:18Z

This pull request was exported from Phabricator. Differential Revision: D27913170

facebook-github-bot · 2021-04-22T02:28:25Z

This pull request was exported from Phabricator. Differential Revision: D27913170

…0d_nccl.py (pytorch#56598) Summary: Pull Request resolved: pytorch#56598 Test Plan: NA Reviewed By: SciPioneer Differential Revision: D27913170 fbshipit-source-id: 4f2c09037b94b37b87545fd2b48ed6502c47433b

facebook-github-bot · 2021-04-22T02:41:03Z

This pull request was exported from Phabricator. Differential Revision: D27913170

facebook-github-bot · 2021-04-22T05:12:09Z

This pull request has been merged in 5cc75e4.

The process group is created on a gloo backend. Context: #56598 (comment) Differential Revision: [D27936805](https://our.internmc.facebook.com/intern/diff/D27936805/) ghstack-source-id: 127150910 Pull Request resolved: #56682

The process group is created on a gloo backend. Context: #56598 (comment) Differential Revision: [D27936805](https://our.internmc.facebook.com/intern/diff/D27936805/) [ghstack-poisoned]

Summary: Pull Request resolved: #56682 The process group is created on a gloo backend. Context: #56598 (comment) ghstack-source-id: 127150910 Test Plan: buck test caffe2/test/distributed:c10d -- test_logging_init Reviewed By: pbelevich Differential Revision: D27936805 fbshipit-source-id: 932efc638f94bdf78ddbae291e3720a20e43f2af

Summary: Fixes test failure after #56598 Introduced by #45335. Pull Request resolved: #57000 Reviewed By: zou3519 Differential Revision: D28030360 Pulled By: seemethere fbshipit-source-id: 4871d51e6b80dceef8bf95c6c658441287575f63

…0d_nccl.py (pytorch#56598) Summary: Pull Request resolved: pytorch#56598 Test Plan: NA Reviewed By: SciPioneer Differential Revision: D27913170 fbshipit-source-id: 3439d18141131b02d55f2ca399a4c795cba2b04b

Summary: Pull Request resolved: pytorch#56682 The process group is created on a gloo backend. Context: pytorch#56598 (comment) ghstack-source-id: 127150910 Test Plan: buck test caffe2/test/distributed:c10d -- test_logging_init Reviewed By: pbelevich Differential Revision: D27936805 fbshipit-source-id: 932efc638f94bdf78ddbae291e3720a20e43f2af

Summary: Fixes test failure after pytorch#56598 Introduced by pytorch#45335. Pull Request resolved: pytorch#57000 Reviewed By: zou3519 Differential Revision: D28030360 Pulled By: seemethere fbshipit-source-id: 4871d51e6b80dceef8bf95c6c658441287575f63

pbelevich requested review from H-Huang, mingzhe09088, mrshenli, pritamdamania87, rohan-varma, wayi1 and zhaojuanmao as code owners April 21, 2021 17:56

facebook-github-bot added oncall: distributed Add this issue/PR to distributed oncall triage queue cla signed labels Apr 21, 2021

facebook-github-bot added the fb-exported label Apr 21, 2021

pbelevich force-pushed the export-D27913170 branch from b3333f0 to 57f0a92 Compare April 21, 2021 19:39

pbelevich force-pushed the export-D27913170 branch from 57f0a92 to e8712cd Compare April 21, 2021 20:05

wayi1 reviewed Apr 21, 2021

View reviewed changes

test/distributed/test_c10d_gloo.py Outdated

Copy link
Copy Markdown

Contributor

wayi1 Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation changed.

wayi1 approved these changes Apr 21, 2021

View reviewed changes

pbelevich force-pushed the export-D27913170 branch from e8712cd to dca061c Compare April 21, 2021 23:41

pbelevich force-pushed the export-D27913170 branch from dca061c to 7b9787c Compare April 22, 2021 02:28

Split test_c10d.py to test_c10d_common.py, test_c10d_gloo.py, test_c1…

751bf5e

…0d_nccl.py (pytorch#56598) Summary: Pull Request resolved: pytorch#56598 Test Plan: NA Reviewed By: SciPioneer Differential Revision: D27913170 fbshipit-source-id: 4f2c09037b94b37b87545fd2b48ed6502c47433b

pbelevich force-pushed the export-D27913170 branch from 7b9787c to 751bf5e Compare April 22, 2021 02:41

facebook-github-bot closed this in 5cc75e4 Apr 22, 2021

facebook-github-bot added the Merged label Apr 22, 2021

wayi1 mentioned this pull request Apr 22, 2021

[c10d] Add requires_gloo decorator to test_logging_init #56682

Closed

skyline75489 mentioned this pull request Apr 27, 2021

Fix path handling on Win32 in rendezvous.py #57000

Closed

Conversation

pbelevich commented Apr 21, 2021

Uh oh!

facebook-github-bot commented Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pytorch_windows_vs2019_py36_cuda10.1_test1 (1/1)

3 failures not recognized by patterns:

🚧 1 ongoing upstream failure:

🚧 1 fixed upstream failure:

Uh oh!

facebook-github-bot commented Apr 21, 2021

Uh oh!

facebook-github-bot commented Apr 21, 2021

Uh oh!

facebook-github-bot commented Apr 21, 2021

Uh oh!

wayi1 commented Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wayi1 Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

pbelevich Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

wayi1 Apr 22, 2021

Choose a reason for hiding this comment

Uh oh!

wayi1 Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wayi1 Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Apr 21, 2021

Uh oh!

facebook-github-bot commented Apr 22, 2021

Uh oh!

facebook-github-bot commented Apr 22, 2021

Uh oh!

facebook-github-bot commented Apr 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

facebook-github-bot commented Apr 21, 2021 •

edited

Loading

wayi1 commented Apr 21, 2021 •

edited

Loading

wayi1 Apr 21, 2021 •

edited

Loading