Thanks, @lukeyeager. Looks good to me.

It's unfortunate that the tool is becoming a pile of backward compat hacks, though. I think a next version of this thing would 1) never prepend Python, and 2) always use the environment.

Temporary workaround 😁

python -m torch.distributed.launch --nproc_per_node=8 --use_env -- -c \
    "import os, sys, subprocess; \
    ret = subprocess.run(['numactl', '--cpunodebind={}'.format(int(int(os.environ['LOCAL_RANK'])/4)), *sys.argv[1:]]); \
    sys.exit(ret.returncode)" python train.py ...

@pietern Thoughts on the more serious proposals above?

Thanks for investigating, @lukeyeager. Unfortunate that we can't have a bool option in argparse...

I like the second option best: a --no_python option and negation in code for readability.

Regarding a v2, we could create a parallel tool called torch.distributed.run start from scratch. Then there won't be any BC issues and we can make it do more stuff as needed. Thoughts?

I like the second option best

Great! Done.

Regarding a v2, we could create a parallel tool called torch.distributed.run start from scratch. Then there won't be any BC issues and we can make it do more stuff as needed. Thoughts?

I don't really have any feelings about that. I haven't been using pytorch much so I don't have much context. I do know there's already torch.nn.parallel.DistributedDataParallel which is (was?) "better" (why?) in some circumstances than torch.distributed.launch. Adding a third thing might be even more confusing? I expect others at NVIDIA might have more well-formed thoughts. I know several people want something which plays nicely with MPI. I would hate for that to become a fourth launch option.

I do know there's already torch.nn.parallel.DistributedDataParallel which is (was?) "better" (why?) in some circumstances than torch.distributed.launch.

They are complementary. You might be thinking of nn.DataParallel (single process, multi GPU, not distributed) and nn.DistributedDataParallel (single process, single or multi GPU, distributed). To launch jobs that use the latter, you need either mpirun, srun, or if you DIY something, you can use torch.distributed.launch. As you see in the code, its job is really simple, and it just launches N processes with a local rank argument / environment variable.

Adding a third thing might be even more confusing? I expect others at NVIDIA might have more well-formed thoughts. I know several people want something which plays nicely with MPI. I would hate for that to become a fourth launch option.

We could have some kind of ptrun frontend for MPI / Slurm / DIY. I know Horovod has hvdrun and does something similar with environment detection and launching a job. This is definitely an area we can improve on.

Thanks for updating the PR.