🚀 The feature, motivation and pitch

Motivation:

• Gives better fault tolerance against hangs in comm init, comm destroy and P2P operations (involves dynamic connection performed by CPU).

• Allow overlap between NCCL init and other stuff users want the main thread to do (e.g., model or data loader init)

Here "non-blocking" refers to whether a NCCL API call would immediately return or block the host CPU (Traditionally, some NCCL APIs such as ncclCommInitRank may block for a little when rendezvous is performed.)

If the user wants the main thread to do other stuff while NCCL is initializing, this mode would also help as it puts NCCL init to the background.

Alternatives

No response

Additional context

This knob is control by PyTorch here:

cc @XilunWu @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o