RpcTestWithSpawn.test_stress_heavy_rpc is flaky in CI #29410
Description
Build: pytorch_macos_10_13_py3_test
Nov 07 12:26:48 ERROR: test_stress_heavy_rpc (__main__.RpcTestWithSpawn)
Nov 07 12:26:48 ----------------------------------------------------------------------
Nov 07 12:26:48 Traceback (most recent call last):
Nov 07 12:26:48 File "/Users/distiller/project/test/common_distributed.py", line 130, in wrapper
Nov 07 12:26:48 self._join_processes(fn)
Nov 07 12:26:48 File "/Users/distiller/project/test/common_distributed.py", line 211, in _join_processes
Nov 07 12:26:48 self._check_return_codes(elapsed_time)
Nov 07 12:26:48 File "/Users/distiller/project/test/common_distributed.py", line 230, in _check_return_codes
Nov 07 12:26:48 raise RuntimeError('Process {} terminated or timed out after {} seconds'.format(i, elapsed_time))
Nov 07 12:26:48 RuntimeError: Process 0 terminated or timed out after 300.0058991909027 seconds
Nov 07 12:26:48
Nov 07 12:26:48 ----------------------------------------------------------------------
Nov 07 12:26:48 Ran 53 tests in 349.255s
cc @ezyang @gchanan @zou3519 @jerryzh168 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @rohan-varma @xush6528