🐛 Bug

Possible root cause for #45435. CC @walterddr
Thanks to @jaglinux for the following triage information.

For barrier call, all reduce uses tensor of device type cuda with the following formula:

int16_t deviceIdx = static_cast<int16_t>(rank_ % numGPUs);

But when the tests are called, the tensors uses different formula to calculate rank to GPU selection.

Hence for rank 0,1,2 ; barrier uses tensors of cuda0, cuda1, cuda2 device type and for testing, tensors of cuda0, cuda2, cuda4 are used.

If we change rank_to_GPU data structure in distributed_test.py to return rank % numGPUs. With this change, all the tests are passing.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd