@vfdev-5 , it seems that HVD test test_idist_all_gather_hvd fails because new_group() does not return a ProcessGroup.NON_GROUP_MEMBER in HVD model in case the rank does not belong to the group, although dist.new_group does.

By the way doesn't HVD really support all_gather with process groups?

https://github.com/horovod/horovod/blob/80780e10d97271472fc6f08cab630ad78d9c5aeb/horovod/torch/mpi_ops.py#L652

By the way doesn't HVD really support all_gather with process groups?

We raise an error for now if a group is passed as arg:

One error in the CI is :

[0]<stderr>:  File "/work/tests/ignite/distributed/utils/__init__.py", line 347, in _test_idist_all_gather_tensors_with_shapes_group
[0]<stderr>:    tensors = all_gather_tensors_with_shapes(rank_tensor, [[r + 2, r + 3, r + 4] for r in ranks], ranks)
[0]<stderr>:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[0]<stderr>:  File "/work/ignite/distributed/utils.py", line 395, in all_gather_tensors_with_shapes
[0]<stderr>:    padded_tensor = torch.nn.functional.pad(
[0]<stderr>:                    ^^^^^^^^^^^^^^^^^^^^^^^^
[0]<stderr>:  File "/opt/conda/lib/python3.11/site-packages/torch/nn/functional.py", line 5096, in pad
[0]<stderr>:    return torch._C._nn.pad(input, pad, mode, value)
[0]<stderr>:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[0]<stderr>:RuntimeError: Padding length should be less than or equal to two times the input dimension but got padding length 6 and input of dimension 1

and we check it with

            with pytest.raises(NotImplementedError, match=r"all_gather with group for horovod is not implemented"):
                tensors = all_gather_tensors_with_shapes(rank_tensor, [[r + 2, r + 3, r + 4] for r in ranks], ranks)

Maybe there is something else is going on ?

In all_gather_tensors_with_shapes, first a group is created, if the arg is given, which is equal to ProcessGroup.NON_GROUP_MEMBER for non-participating processes. But for HVD, the resulting group is a ProcessSet anyway. When padding is being done, which takes places before allgather, it tries to pad the 1-dim input tensor which is torch.tensor([NON_PARITIPATING_RANK]) with the padding size argument of 3 dimensions (3 * 2 (for left and right padding)) and raises error.

I mean the below if statement is not enough for horovod as the resulting group is always a ProcessSet

I mean the below if statement is not enough for horovod as the resulting group is always a ProcessSet

ignite/ignite/distributed/utils.py

Lines 390 to 391 in ed934ef

	if isinstance(_model, _SerialModel) or group == dist.GroupMember.NON_GROUP_MEMBER:
	return [tensor]

actually, this is weird why we use specific dist.GroupMember.NON_GROUP_MEMBER in the generic all_gather_tensors_with_shapes method...

So we could replace it with something like this maybe:

# In idist:

def is_in_group(group):
    if group is dist.GroupMember.NON_GROUP_MEMBER:
        return False
    if isinstance(group, hvd.ProcessSet):
        return group.included()
    return True

So we could replace it with something like this maybe:

# In idist:

def is_in_group(group):
    if group is dist.GroupMember.NON_GROUP_MEMBER:
        return False
    if isinstance(group, hvd.ProcessSet):
        return group.included()
    return True

Yes, this makes sense. I would just make it private and call like in pytorch: _rank_not_in_group.
Sadra, can you please send a PR with this update?

I think we can close this PR