Skip to content

Disable test_distributed_for for multigpu test env #47703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrshenli wants to merge 2 commits into from
Closed

Disable test_distributed_for for multigpu test env #47703

mrshenli wants to merge 2 commits into from

Conversation

@mrshenli
Copy link
Contributor

@mrshenli mrshenli commented Nov 10, 2020 •
edited
Loading

Stack from ghstack:

Differential Revision: D24871454

mrshenli added a commit that referenced this pull request Nov 10, 2020
@mrshenli
Use test_distributed_spawn for multigpu test
6508688 
ghstack-source-id: c05994f
Pull Request resolved: #47703
@mrshenli mrshenli requested a review from rohan-varma November 10, 2020 22:32
@mrshenli
Copy link
Contributor Author

mrshenli commented Nov 10, 2020

ci-all test in #47639

@mrshenli mrshenli changed the title Use test_distributed_spawn for multigpu test Disable test_distributed_for for multigpu test env Nov 10, 2020
mrshenli added a commit that referenced this pull request Nov 10, 2020
@mrshenli
Disable test_distributed_for for multigpu test env
6bd19e6 
ghstack-source-id: 9ea8b83
Pull Request resolved: #47703
@dr-ci
Copy link

dr-ci bot commented Nov 10, 2020 •
edited
Loading

💊 CI failures summary and remediations

As of commit d296461 (more details on the Dr. CI page):

  • 4/4 failures possibly* introduced in this PR
    • 1/4 non-CircleCI failure(s)

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_xla_linux_bionic_py3_6_clang9_test (1/3)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 10 23:59:53 sccache: error: couldn't connect to server 
Nov 10 23:59:53 +++ eval 'extract_trap_cmd ' 
Nov 10 23:59:53 ++++ extract_trap_cmd 
Nov 10 23:59:53 ++++ printf '%s\n' '' 
Nov 10 23:59:53 +++ printf '%s\n' cleanup 
Nov 10 23:59:53 ++ trap -- ' 
Nov 10 23:59:53 cleanup' EXIT 
Nov 10 23:59:53 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test != *pytorch-win-* ]] 
Nov 10 23:59:53 ++ which sccache 
Nov 10 23:59:53 ++ sccache --stop-server 
Nov 10 23:59:53 Stopping sccache server... 
Nov 10 23:59:53 sccache: error: couldn't connect to server 
Nov 10 23:59:53 sccache: caused by: Connection refused (os error 111) 
Nov 10 23:59:53 ++ true 
Nov 10 23:59:53 ++ rm /var/lib/jenkins/sccache_error.log 
Nov 10 23:59:53 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test == *rocm* ]] 
Nov 10 23:59:53 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 
Nov 10 23:59:53 ++ SCCACHE_IDLE_TIMEOUT=1200 
Nov 10 23:59:53 ++ RUST_LOG=sccache::server=error 
Nov 10 23:59:53 ++ sccache --start-server 
Nov 10 23:59:53 sccache: Starting the server... 
Nov 10 23:59:53 ++ sccache --zero-stats

See CircleCI build pytorch_linux_backward_compatibility_check_test (2/3)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 10 23:30:48 The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
Nov 10 23:30:48 processing existing schema:  __getstate__(__torch__.torch.classes.xnnpack.LinearOpContext _0) -> ((Tensor, Tensor?, Scalar?, Scalar?) _0) 
Nov 10 23:30:48 processing existing schema:  __setstate__(__torch__.torch.classes.xnnpack.LinearOpContext _0, (Tensor, Tensor?, Scalar?, Scalar?) _1) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  __getstate__(__torch__.torch.classes.xnnpack.Conv2dOpContext _0) -> ((Tensor, Tensor?, int[], int[], int[], int, Scalar?, Scalar?) _0) 
Nov 10 23:30:48 processing existing schema:  __setstate__(__torch__.torch.classes.xnnpack.Conv2dOpContext _0, (Tensor, Tensor?, int[], int[], int[], int, Scalar?, Scalar?) _1) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  __getstate__(__torch__.torch.classes.xnnpack.TransposeConv2dOpContext _0) -> ((Tensor, Tensor?, int[], int[], int[], int[], int, Scalar?, Scalar?) _0) 
Nov 10 23:30:48 processing existing schema:  __setstate__(__torch__.torch.classes.xnnpack.TransposeConv2dOpContext _0, (Tensor, Tensor?, int[], int[], int[], int[], int, Scalar?, Scalar?) _1) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  __init__(__torch__.torch.classes._nnapi.Compilation _0) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  init(__torch__.torch.classes._nnapi.Compilation _0, Tensor _1, Tensor[] _2) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  run(__torch__.torch.classes._nnapi.Compilation _0, Tensor[] _1, Tensor[] _2) -> (None _0) 
Nov 10 23:30:48 processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (None _0) 
Nov 10 23:30:48 The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not.  
Nov 10 23:30:48  
Nov 10 23:30:48 Broken ops: [ 
Nov 10 23:30:48 	aten::_foreach_log(Tensor[] tensors) -> (Tensor[]) 
Nov 10 23:30:48 	aten::_foreach_round(Tensor[] tensors) -> (Tensor[]) 
Nov 10 23:30:48 	aten::_foreach_sinh(Tensor[] tensors) -> (Tensor[]) 
Nov 10 23:30:48 	aten::_foreach_lgamma_(Tensor[] self) -> () 
Nov 10 23:30:48 	aten::_foreach_lgamma(Tensor[] tensors) -> (Tensor[]) 
Nov 10 23:30:48 	aten::_foreach_log10(Tensor[] tensors) -> (Tensor[]) 
Nov 10 23:30:48 	aten::_foreach_round_(Tensor[] self) -> () 
Nov 10 23:30:48 	aten::_foreach_sin(Tensor[] tensors) -> (Tensor[])

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_test (3/3)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 10 23:30:14 sccache: error: couldn't connect to server 
Nov 10 23:30:14 +++ eval 'extract_trap_cmd ' 
Nov 10 23:30:14 ++++ extract_trap_cmd 
Nov 10 23:30:14 ++++ printf '%s\n' '' 
Nov 10 23:30:14 +++ printf '%s\n' cleanup 
Nov 10 23:30:14 ++ trap -- ' 
Nov 10 23:30:14 cleanup' EXIT 
Nov 10 23:30:14 ++ [[ pytorch-linux-xenial-py3.6-gcc5.4-test != *pytorch-win-* ]] 
Nov 10 23:30:14 ++ which sccache 
Nov 10 23:30:14 ++ sccache --stop-server 
Nov 10 23:30:14 Stopping sccache server... 
Nov 10 23:30:14 sccache: error: couldn't connect to server 
Nov 10 23:30:14 sccache: caused by: Connection refused (os error 111) 
Nov 10 23:30:14 ++ true 
Nov 10 23:30:14 ++ rm /var/lib/jenkins/sccache_error.log 
Nov 10 23:30:14 ++ [[ pytorch-linux-xenial-py3.6-gcc5.4-test == *rocm* ]] 
Nov 10 23:30:14 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 
Nov 10 23:30:14 ++ SCCACHE_IDLE_TIMEOUT=1200 
Nov 10 23:30:14 ++ RUST_LOG=sccache::server=error 
Nov 10 23:30:14 ++ sccache --start-server 
Nov 10 23:30:14 sccache: Starting the server... 
Nov 10 23:30:14 ++ sccache --zero-stats

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 3 times.

@mrshenli
Copy link
Contributor Author

mrshenli commented Nov 11, 2020

test failures are irrelevant, landing:

Nov 10 23:50:23 Executing ['/opt/conda/bin/python', 'test_xnnpack_integration.py', '--verbose'] ... [2020-11-10 23:50:23.688878]
Nov 10 23:50:24 Traceback (most recent call last):
Nov 10 23:50:24   File "test_xnnpack_integration.py", line 8, in <module>
Nov 10 23:50:24     import torch.testing._internal.hypothesis_utils as hu
Nov 10 23:50:24   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/hypothesis_utils.py", line 6, in <module>
Nov 10 23:50:24     import hypothesis
Nov 10 23:50:24 ModuleNotFoundError: No module named 'hypothesis'

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Nov 11, 2020

@mrshenli merged this pull request in 1b95474.

@facebook-github-bot facebook-github-bot deleted the gh/mrshenli/262/head branch November 14, 2020 15:17
mrshenli added a commit that referenced this pull request Nov 16, 2020
@mrshenli
Revert "Disable test_distributed_for for multigpu test env (#47703)"
38b7748 
This reverts commit 1b95474.
@mrshenli mrshenli mentioned this pull request Nov 16, 2020
Closed
mrshenli added a commit that referenced this pull request Nov 16, 2020
@mrshenli
Revert "Disable test_distributed_for for multigpu test env (#47703)"
abe8fbb 
This reverts commit 1b95474.

[ghstack-poisoned]
mrshenli added a commit that referenced this pull request Nov 16, 2020
@mrshenli
Revert "Disable test_distributed_for for multigpu test env (#47703)"
130bc1b 
This reverts commit 1b95474.

ghstack-source-id: ced51f1
Pull Request resolved: #48026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@rohan-varma rohan-varma rohan-varma approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mrshenli @facebook-github-bot @rohan-varma