Add docs for RPC, dist autograd, and RRef modules #29276

rohan-varma · 2019-11-06T06:00:15Z

Closes #28983. Documentation for torch.distributed.rpc and torch.distributed.autograd modules. Also fixes/tidies up some of the docstrings in rpc/autograd, and moves some functions to be private so they don't show up in the documentation.

Note: Much of the text to describe/explain the RPC/RRef layers are taken from the following RFCs: #23110, #26759

pritamdamania87 · 2019-11-08T19:14:41Z

docs/source/model_parallel.rst

+.. automodule:: torch.distributed.rpc
+.. currentmodule:: torch.distributed.rpc
+
+.. autofunction:: rpc_sync


Is there a way to automatically pull in all public methods of a module instead of specifying each one like this?

@pritamdamania87 it looks like other doc-related commits list out each public method, see for example 8915e27#diff-7aa5f4b7f9dedcb7a6ce129b9d3c5eb8. The sphix documentation (http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) mentions we can recursively include members of a class/module, which I've updated this PR to use.

pritamdamania87 · 2019-11-08T19:15:48Z

docs/source/model_parallel.rst

@@ -0,0 +1,11 @@
+.. role:: hidden


Is this hidden on purpose? I think it should be fine to not hide this for now since it'll only be in master and won't show up in some stable docs.

Should we keep them hidden if the docs aren't polished completely?

(Also in response to similar question from @mrshenli)
As @pritamdamania mentioned, it's fine to have unfinished docs out in master; however, tagging them as hidden can be a helpful way to keep track of what still needs work prior to the final release.

pritamdamania87 · 2019-11-08T19:16:19Z

docs/source/model_parallel.rst

+.. role:: hidden
+    :class: hidden-section
+
+Distributed RPC Framework - torch.distributed.autograd and torch.distributed.rpc


Only Distributed RPC Framework should be sufficient, otherwise the entire title is too long for the title.

mrshenli

Thanks for adding this!

Hey @jlin27, should we keep this hidden for now as we haven't finished polishing the docstrings yet?

mrshenli · 2019-11-11T18:09:58Z

docs/source/model_parallel.rst

+RPC and RRef Framework
+====================================================
+
+Bbasics


mrshenli · 2019-11-11T18:11:22Z

torch/distributed/rpc/api.py

+        Returns the result of running ``func`` on ``args`` and ``kwargs``.

    Example::
+


Curious: why do we need the new line here?

Adding this makes the code appear nicely and in python format on the documentation page

rohan-varma · 2019-11-11T18:24:42Z

docs/source/model_parallel.rst

+Before using RPC and distributed autograd primitives, initialization must take place. First, a backend over which RPCs can be sent over must be initialized. The default (and currently, only available) implementation is the `ProcessGroup` backend, and must be initialized with `torch.distributed.init_process_group` before using other function. See the `documentation for torch.distributed <https://pytorch.org/docs/stable/distributed.html>`_ for additional details. Next, the local RPC agent can be initialized, after which the process will be able to send and receive RPCs from all other connected processes.
+
+.. automodule:: torch.distributed.rpc
+    :members:


@pritamdamania87 this will add all members of this module, but we'd want to be careful in using this since non-polished/private functions could show up.

docs/source/rpc.rst

torch/csrc/distributed/rpc/init.cpp

mrshenli

The content LGTM! Please also get a stamp from @jlin27 on whether we should hide this for now and whether the structure is OK. Thanks!

docs/source/rpc.rst

pritamdamania87 · 2019-11-11T20:27:01Z

docs/source/rpc.rst

+Distributed RPC Framework
+=====================================================
+
+The distributed RPC framework provides mechanisms for multi-machine model training through a set of primitives to allow for remote communication, and a higher-level API to automatically differentiate models split across several machines.


Why do we have everything on a single long line? Can we split the lines in this file? Looks like most doc files split lines after 80 chars.

docs/source/rpc.rst

rohan-varma · 2019-11-12T00:31:15Z

@pritamdamania87 All the comments are addressed, could you take another look? Thanks!

docs/source/rpc.rst

pritamdamania87

Looks good to me, although please check why the RRef class doesn't show up and see if we can fix it before landing.

rohan-varma · 2019-11-12T19:39:50Z

@jlin27 Could you please take a look at the structure and whether it should be hidden before I land this? Thanks!

facebook-github-bot

@rohan-varma has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@rohan-varma has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

mrshenli · 2019-11-13T19:57:53Z

docs/source/rpc.rst

+.. role:: hidden
+    :class: hidden-section
+
+Distributed RPC Framework


Let's mark this as experimental for now. Check out this page as an example.

Will do this in follow up PR

rohan-varma · 2019-11-14T17:02:04Z

All of the failures are pre existing and not related to this PR. Proceeding with landing.

facebook-github-bot · 2019-11-14T23:48:43Z

@rohan-varma merged this pull request in 06ef4a7.

Summary: Closes pytorch#28983. Documentation for `torch.distributed.rpc` and `torch.distributed.autograd` modules. Also fixes/tidies up some of the docstrings in rpc/autograd, and moves some functions to be private so they don't show up in the documentation. Note: Much of the text to describe/explain the RPC/RRef layers are taken from the following RFCs: pytorch#23110, pytorch#26759 Pull Request resolved: pytorch#29276 Differential Revision: D18478754 Pulled By: rohan-varma fbshipit-source-id: e9a7089baf5275304e5408d319eb9bf98e53fff8

rohan-varma added 2 commits November 5, 2019 21:57

Starting RPC and dist autograd docs

9020afb

remove testing comment

daf30c9

rohan-varma added module: rpc Related to RPC, distributed autograd, RRef, and distributed optimizer module: docs Related to our documentation, both in docs/ and docblocks labels Nov 6, 2019

pritamdamania87 reviewed Nov 8, 2019

View reviewed changes

Add RPC, RRef, dist autograd docs

c156726

rohan-varma requested review from apaszke, mrshenli and pietern as code owners November 11, 2019 04:56

mrshenli reviewed Nov 11, 2019

View reviewed changes

mrshenli requested a review from jlin27 November 11, 2019 18:12

More updates

8016710

rohan-varma changed the title ~~[WIP][docs] distributed framework docs~~ [docs] distributed framework docs Nov 11, 2019

rohan-varma changed the title ~~[docs] distributed framework docs~~ [docs] Add docs for RPC, dist autograd, and RRef modules Nov 11, 2019

rohan-varma commented Nov 11, 2019

View reviewed changes

rohan-varma added 3 commits November 11, 2019 10:26

rename

9a6fc02

Add docsstring for RRef class

a4954ce

Move more functions to private

c74d1e9

mrshenli reviewed Nov 11, 2019

View reviewed changes

docs/source/rpc.rst Outdated Show resolved Hide resolved

docs/source/rpc.rst Outdated Show resolved Hide resolved

torch/csrc/distributed/rpc/init.cpp Outdated Show resolved Hide resolved

mrshenli approved these changes Nov 11, 2019

View reviewed changes

pritamdamania87 mentioned this pull request Nov 11, 2019

Design doc for distributed autograd. #29175

Closed

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

ddab505

pritamdamania87 reviewed Nov 11, 2019

View reviewed changes

rohan-varma added 2 commits November 11, 2019 16:09

Address comments

7fc3fa3

Minor cosmetic updates

528008d

rohan-varma changed the title ~~[docs] Add docs for RPC, dist autograd, and RRef modules~~ Add docs for RPC, dist autograd, and RRef modules Nov 12, 2019

rohan-varma added 3 commits November 11, 2019 16:19

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

40518cb

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

b41a905

Format code example

9d0fd8a

rohan-varma requested a review from pritamdamania87 November 12, 2019 00:31

rohan-varma commented Nov 12, 2019

View reviewed changes

docs/source/rpc.rst Show resolved Hide resolved

pritamdamania87 approved these changes Nov 12, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

48cef21

facebook-github-bot reviewed Nov 13, 2019

View reviewed changes

rohan-varma added 2 commits November 13, 2019 10:34

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

d2c2159

Merge remote-tracking branch 'origin/master' into rpc_autograd_docs

8b8e66e

facebook-github-bot reviewed Nov 13, 2019

View reviewed changes

mrshenli reviewed Nov 13, 2019

View reviewed changes

facebook-github-bot closed this in 06ef4a7 Nov 14, 2019

facebook-github-bot added the merged label Nov 14, 2019

facebook-github-bot deleted the rpc_autograd_docs branch July 13, 2020 17:57

mruberry added the Merged label Oct 28, 2020

		Returns the result of running ``func`` on ``args`` and ``kwargs``.

		Example::

Add docs for RPC, dist autograd, and RRef modules #29276

Add docs for RPC, dist autograd, and RRef modules #29276

Uh oh!

Conversation

rohan-varma commented Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohan-varma Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrshenli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrshenli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohan-varma commented Nov 12, 2019

Uh oh!

Uh oh!

pritamdamania87 left a comment

Choose a reason for hiding this comment

Uh oh!

rohan-varma commented Nov 12, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohan-varma commented Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Nov 14, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

rohan-varma commented Nov 6, 2019 •

edited

Loading

rohan-varma Nov 11, 2019 •

edited

Loading

rohan-varma commented Nov 14, 2019 •

edited

Loading