It's odd to have init_model_parallel under "rpc".

it should be like

import torch.distributed.model_parallel as model_parallel

model_parallel.init_model_parallel(...)
model_parallel.rpc_sync(...)

@mrshenli What's your opion?

It's odd to have init_model_parallel under "rpc".

I agree. Probably we don't need to use the phrase "model parallel" here. @aazzolini has a point that "model parallel" does not fully cover the scope of this feature. For example, torch.distributed.rpc, can support data parallel and parameter server as well.

This init method here initializes contexts and setup communication channels if necessary. After this point, the application could use it for distributed training, and might use it for other purposes as well. So, maybe, the name does not have to imply what the user would like to do next.

Shall we call it torch.distributed.rpc.init_worker instead? The sync API can also be named as torch.distributed.rpc.sync_worker, but let's add sync_worker in a followup PR after we address #27096.

Shall we call it torch.distributed.rpc.init_worker instead? The sync API can also be named as torch.distributed.rpc.sync_worker, but let's add sync_worker in a followup PR after we address #27096.

I'm assuming the proposal here is that we will still have init_model_parallel in a different package that would call rpc.init_worker underneath?

I'm assuming the proposal here is that we will still have init_model_parallel in a different package that would call rpc.init_worker underneath?

No, I was actually proposing to replace init_model_parallel with init_worker. Say we provide an init_model_parallel training, and an application would like to use rpc and distributed autograd to do data parallel training, it would look weird that the application has to call init_model_parallel.

Say we provide an init_model_parallel training, and an application would like to use rpc and distributed autograd to do data parallel training, it would look weird that the application has to call init_model_parallel.

If you are using distributed autograd, doesn't that imply model parallel? Distributed autograd comes into play only if a model is split across multiple machines.

I don't think "model parallel" should even be a word anywhere in pytorch.distributed . Model parallelism is a charged term that we don't really need in order to convey what the rpc package does. RRef is a natural consequence of being able to run remote "methods" (as opposed to functions), and distributed autograd is a natural consequence of being able to pass tensors with require_grad=True over RPC. None of this needs to imply that a model even exists, here.

Model parallel only makes sense if you have a model, so it's more tied to the concept of nn.Module etc.

So I really think that init_model_parallel doesn't need to exist as a function anywheree. instead, call it rpc.init_worker() or even rpc.init().

(As a side note, at some point soon it would be better if we closed on a final API since these changes will actually break some of the stuff we have built on top of it already).

I agree -- init_model_parallel is too broad here, since we're only initializing the RPC.

This this PR only moves the stuff around, let's stack another PR on top that renames (and adds a few lines for backward compat for everything that's in flight today).