Let init_process_group block until all peers are ready #45126

mrshenli · 2020-09-22T14:10:34Z

Stack from ghstack:

Let init_process_group block until all peers are ready #45126 Let init_process_group block until all peers are ready

Differential Revision: D23841218

[ghstack-poisoned]

ghstack-source-id: ee9d8ab Pull Request resolved: #45126

dr-ci · 2020-09-22T14:58:11Z

💊 CI failures summary and remediations

As of commit e3e36b0 (more details on the Dr. CI page):

7/8 failures possibly* introduced in this PR
- 1/7 non-CircleCI failure(s)
1/8 broken upstream at merge base 1b059f2 since Sep 22

🕵️ 6 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_6_gcc5_4_test (1/6)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 22 15:03:34 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" }

Sep 22 15:03:34 Traceback (most recent call last): 
Sep 22 15:03:34   File "test/run_test.py", line 741, in <module> 
Sep 22 15:03:34     main() 
Sep 22 15:03:34   File "test/run_test.py", line 724, in main 
Sep 22 15:03:34     raise RuntimeError(err_message) 
Sep 22 15:03:34 RuntimeError: distributed/test_distributed_fork failed! 
Sep 22 15:03:34 =================== sccache compilation log =================== 
Sep 22 15:03:34 + cleanup 
Sep 22 15:03:34 + retcode=1 
Sep 22 15:03:34 + set +x 
Sep 22 15:03:34 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Sep 22 15:03:34  
Sep 22 15:03:34 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Sep 22 15:03:34 Compile requests                 61 
Sep 22 15:03:34 Compile requests executed        34 
Sep 22 15:03:34 Cache hits                       26 
Sep 22 15:03:34 Cache misses                      7 
Sep 22 15:03:34 Cache timeouts                    0 
Sep 22 15:03:34 Cache read errors                 0 
Sep 22 15:03:34 Forced recaches                   0 
Sep 22 15:03:34 Cache write errors                0

pytorch_macos_10_13_py3_test (2/6)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Sep 22 07:55:09 ERROR [13.994s]: test_backend_apis (__main__.TestBackendDynamicLoad)

Sep 22 07:54:54 MPI not available -- MPI backend tests will be skipped 
Sep 22 07:54:54 Running distributed tests for the test backendwith env init_method 
Sep 22 07:54:54 Executing ['/Users/distiller/workspace/miniconda3/bin/python', 'distributed/test_distributed_fork.py', '--verbose'] ... [2020-09-22 07:54:54.118783] 
Sep 22 07:54:55 Test results will be stored in test-reports/python-unittest 
Sep 22 07:54:55  
Sep 22 07:54:55 Running tests... 
Sep 22 07:54:55 ---------------------------------------------------------------------- 
Sep 22 07:55:09   test_backend_apis (__main__.TestBackendDynamicLoad) ... ERROR (13.994s) 
Sep 22 07:55:09  
Sep 22 07:55:09 ====================================================================== 
Sep 22 07:55:09 ERROR [13.994s]: test_backend_apis (__main__.TestBackendDynamicLoad) 
Sep 22 07:55:09 ---------------------------------------------------------------------- 
Sep 22 07:55:09 Traceback (most recent call last): 
Sep 22 07:55:09   File "distributed/test_distributed_fork.py", line 39, in wrapper 
Sep 22 07:55:09     return func(*args, **kwargs) 
Sep 22 07:55:09   File "distributed/test_distributed_fork.py", line 83, in test_backend_apis 
Sep 22 07:55:09     dist.init_process_group(backend='test', init_method='env://', world_size=1, rank=0) 
Sep 22 07:55:09   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 438, in init_process_group 
Sep 22 07:55:09     _default_pg.barrier() 
Sep 22 07:55:09 RuntimeError: ProcessGroupTest does not support barrier 
Sep 22 07:55:09

pytorch_linux_bionic_py3_6_clang9_test (3/6)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 22 14:58:38 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" }

Sep 22 14:58:38     raise RuntimeError(err_message) 
Sep 22 14:58:38 RuntimeError: distributed/test_distributed_fork failed! 
Sep 22 14:58:38  
Sep 22 14:58:38 real	9m25.768s 
Sep 22 14:58:38 user	9m4.752s 
Sep 22 14:58:38 sys	1m46.399s 
Sep 22 14:58:38 + cleanup 
Sep 22 14:58:38 + retcode=1 
Sep 22 14:58:38 + set +x 
Sep 22 14:58:38 =================== sccache compilation log =================== 
Sep 22 14:58:38 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Sep 22 14:58:38  
Sep 22 14:58:38 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Sep 22 14:58:38 Compile requests                 61 
Sep 22 14:58:38 Compile requests executed        34 
Sep 22 14:58:38 Cache hits                       26 
Sep 22 14:58:38 Cache misses                      7 
Sep 22 14:58:38 Cache timeouts                    0 
Sep 22 14:58:38 Cache read errors                 0 
Sep 22 14:58:38 Forced recaches                   0 
Sep 22 14:58:38 Cache write errors                0

pytorch_linux_bionic_py3_8_gcc9_coverage_test (4/6)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 22 15:04:33 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:22: error: expected ‘;’ before ‘}’ token\n    2 | int main() { return 0 }\n      |                      ^~\n      |                      ;\n" }

Sep 22 15:04:33     raise RuntimeError(err_message) 
Sep 22 15:04:33 RuntimeError: distributed/test_distributed_fork failed! 
Sep 22 15:04:33  
Sep 22 15:04:33 real	9m40.825s 
Sep 22 15:04:33 user	9m56.628s 
Sep 22 15:04:33 sys	0m57.194s 
Sep 22 15:04:33 + cleanup 
Sep 22 15:04:33 + retcode=1 
Sep 22 15:04:33 + set +x 
Sep 22 15:04:33 =================== sccache compilation log =================== 
Sep 22 15:04:33 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:22: error: expected ‘;’ before ‘}’ token\n    2 | int main() { return 0 }\n      |                      ^~\n      |                      ;\n" } 
Sep 22 15:04:33  
Sep 22 15:04:33 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Sep 22 15:04:33 Compile requests                 61 
Sep 22 15:04:33 Compile requests executed        34 
Sep 22 15:04:33 Cache hits                       26 
Sep 22 15:04:33 Cache misses                      7 
Sep 22 15:04:33 Cache timeouts                    0 
Sep 22 15:04:33 Cache read errors                 0 
Sep 22 15:04:33 Forced recaches                   0 
Sep 22 15:04:33 Cache write errors                0

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (5/6)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 22 15:02:27 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" }

Sep 22 15:02:27 Traceback (most recent call last): 
Sep 22 15:02:27   File "test/run_test.py", line 741, in <module> 
Sep 22 15:02:27     main() 
Sep 22 15:02:27   File "test/run_test.py", line 724, in main 
Sep 22 15:02:27     raise RuntimeError(err_message) 
Sep 22 15:02:27 RuntimeError: distributed/test_distributed_fork failed! 
Sep 22 15:02:27 + cleanup 
Sep 22 15:02:27 + retcode=1 
Sep 22 15:02:27 + set +x 
Sep 22 15:02:27 =================== sccache compilation log =================== 
Sep 22 15:02:27 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Sep 22 15:02:27  
Sep 22 15:02:27 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Sep 22 15:02:27 Compile requests                 61 
Sep 22 15:02:27 Compile requests executed        34 
Sep 22 15:02:27 Cache hits                       26 
Sep 22 15:02:27 Cache misses                      7 
Sep 22 15:02:27 Cache timeouts                    0 
Sep 22 15:02:27 Cache read errors                 0 
Sep 22 15:02:27 Forced recaches                   0 
Sep 22 15:02:27 Cache write errors                0

pytorch_linux_xenial_py3_clang5_asan_test2 (6/6)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 22 14:44:31 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:11:3 in

Sep 22 14:44:31     #7 0x5610858ac70b in PyEval_EvalCode /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:731 
Sep 22 14:44:31     #8 0x56108592c573 in run_mod /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:1025 
Sep 22 14:44:31     #9 0x56108592c60c in PyRun_StringFlags /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:949 
Sep 22 14:44:31     #10 0x56108592c66e in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:445 
Sep 22 14:44:31     #11 0x561085930472 in run_command /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:301 
Sep 22 14:44:31     #12 0x561085930472 in Py_Main /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:749 
Sep 22 14:44:31     #13 0x5610857fa43d in main /tmp/build/80754af9/python_1599604603603/work/Programs/python.c:69 
Sep 22 14:44:31     #14 0x7f5835b0683f in __libc_start_main /build/glibc-e6zv40/glibc-2.23/csu/../csu/libc-start.c:291 
Sep 22 14:44:31     #15 0x5610858d9d0a in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103 
Sep 22 14:44:31  
Sep 22 14:44:31 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:11:3 in  
Sep 22 14:44:31 + retcode=1 
Sep 22 14:44:31 + set -e 
Sep 22 14:44:31 + return 1 
Sep 22 14:44:31 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 == *-NO_AVX-* ]] 
Sep 22 14:44:31 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 == *-NO_AVX2-* ]] 
Sep 22 14:44:31 + '[' -n https://github.com/pytorch/pytorch/pull/45126 ']' 
Sep 22 14:44:31 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 != *coverage* ]] 
Sep 22 14:44:31 ++ mktemp 
Sep 22 14:44:31 + DETERMINE_FROM=/tmp/tmp.ACo531zUF6 
Sep 22 14:44:31 + file_diff_from_base /tmp/tmp.ACo531zUF6

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet:

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test since Sep 22
- 🔁 rerun

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.7-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 4 times.

mrshenli · 2020-09-22T15:35:55Z

Tried this on MacOS. This is not sufficient to fix the error.

mrshenli · 2020-09-23T01:20:31Z

superseded by #45181

Let init_process_group block until all peers are ready

e3e36b0

[ghstack-poisoned]

mrshenli requested review from apaszke, pietern, pritamdamania87, rohan-varma and zhaojuanmao as code owners September 22, 2020 14:10

mrshenli added a commit that referenced this pull request Sep 22, 2020

Let init_process_group block until all peers are ready

5bee150

ghstack-source-id: ee9d8ab Pull Request resolved: #45126

mrshenli mentioned this pull request Sep 22, 2020

Fix RPC and ProcessGroup GIL deadlock #45088

Closed

mrshenli mentioned this pull request Sep 23, 2020

Add barrier() at the end of init_process_group and new_group. #45181

Closed

mrshenli closed this Sep 23, 2020

facebook-github-bot deleted the gh/mrshenli/242/head branch October 23, 2020 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Let init_process_group block until all peers are ready #45126

Let init_process_group block until all peers are ready #45126

Uh oh!

mrshenli commented Sep 22, 2020 •

edited

Loading

Uh oh!

dr-ci bot commented Sep 22, 2020 •

edited

Loading

Uh oh!

mrshenli commented Sep 22, 2020

Uh oh!

mrshenli commented Sep 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Let init_process_group block until all peers are ready #45126

Let init_process_group block until all peers are ready #45126

Uh oh!

Conversation

mrshenli commented Sep 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Sep 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 6 new failures recognized by patterns

pytorch_linux_xenial_py3_6_gcc5_4_test (1/6)

pytorch_macos_10_13_py3_test (2/6)

pytorch_linux_bionic_py3_6_clang9_test (3/6)

pytorch_linux_bionic_py3_8_gcc9_coverage_test (4/6)

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (5/6)

pytorch_linux_xenial_py3_clang5_asan_test2 (6/6)

🚧 1 ongoing upstream failure:

ci.pytorch.org: 1 failed

Uh oh!

mrshenli commented Sep 22, 2020

Uh oh!

mrshenli commented Sep 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mrshenli commented Sep 22, 2020 •

edited

Loading

dr-ci bot commented Sep 22, 2020 •

edited

Loading