In the implementation for the backward pass in #27022, we accumulate the gradients in the autograd context by default. In order to have symmetry with torch.autograd.backwards, the API should be something like this:

torch.distributed.autograd.backward(tensors, grads, accumulate_grad_on_tensors = True)

When accumulate_grad_on_tensors is set to False, we accumulate the grads on the autograd context, otherwise we accumulate the grads on the .grad field.

cc @ezyang @ssnl @albanD @zou3519 @gqchen @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @aazzolini @xush6528