🚀 Feature

Idea is to provide support for Horovod distributed configuration in the same way as it is done for native pytorch dist and xla.