Hi @IsaacYangSLA and @wyli ,

I am trying to add distributed tests based on Horovod environment, but I got a question here:
As Horovod needs to execute its own command in shell to launch the program, how can we launch it in CI automatically?
For example: horovodrun -np 2 python test_evenly_divisible_all_gather_hvd.py
I checked our current distributed tests, seems it uses torch.multiprocessing.Process to launch in a python program instead of runningtorch.dist.launch command.

Thanks in advance.

/black

Hi @wyli @IsaacYangSLA , or maybe let's just put this Horovod test in the tests to manually run locally first, then try to integrate into CI later when we have more Horovod tests? Just like what we did for the PyTorch distributed tests before?

Thanks.

To run this new Horovod test locally, just follow:

1. Install Horovod:
   HOROVOD_NCCL_INCLUDE=/usr/include HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu HOROVOD_GPU_OPERATIONS=NCCL \ HOROVOD_NCCL_LINK=SHARED pip install --no-cache-dir horovod
2. Execute on 2 GPUs in a single machine:
   horovodrun -np 2 python test_evenly_divisible_all_gather_hvd.py

Thanks.

@IsaacYangSLA could you please help review?