Skip to content

Python/C++ API Parity: torch.nn modules and functional #25883

New issue
New issue
Open
Open
Python/C++ API Parity: torch.nn modules and functional#25883
Assignees
glaringlee
Labels
good first issuemodule: cppRelated to C++ APImodule: nnRelated to torch.nntriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
@yf225

Description

@yf225
yf225
opened on Sep 9, 2019

Currently, PyTorch C++ API is missing many torch::nn layers that are available in the Python API. As part of the Python/C++ API parity work, we would like to add the following torch::nn modules and utilities in C++ API:

Containers

Convolution layers

Pooling layers

Padding layers

Non-linear activations (weighted sum, nonlinearity)

Non-linear activations (other)

Normalization layers

Recurrent layers

Transformer layers

Linear layers

Dropout layers

Sparse layers

Distance functions

Loss functions

Vision layers

Utilities

torch.nn.functional

Implementation Notes:

  • For torch.nn modules, Python and C++ implementation must only differ in language-specific syntax, and their data member fields, control flow and logic must be exactly the same.
    • You might see that some of the torch.nn modules call the corresponding torch.nn.functional functions. When implementing the C++ version of those modules, we should also add the corresponding torch::nn::functional functions, in order to preserve the call structure.
  • When you are adding a new C++ torch::nn module:
    • The new torch::nn module must subclass from public Cloneable<ModuleName>. For example, class TORCH_API LinearImpl : public Cloneable<LinearImpl> (in torch/csrc/api/include/torch/nn/modules/linear.h).
    • How do we decide which file to put the new C++ torch::nn module in? Answer: We should look at where the Python version of module is located and try to mirror that file structure. For example, the Python version of torch.nn.Linear lives in torch/nn/modules/linear.py, so the C++ version torch::nn::Linear should live in torch/csrc/api/include/torch/nn/modules/linear.h.
    • If you add a new module header, you must also add the include statement for the new header in torch/csrc/api/include/torch/nn/modules.h.
    • If you add a new module .cpp file, you must also add it into caffe2/CMakeLists.txt and tools/build_variables.py. (Please search for other modules in those files to see how this should be done.)
    • You must add tests for the new module in test/cpp/api/modules.cpp. In particular, make sure the module's pretty_print is tested and it outputs the same value as the Python version.
    • if the module is templatized (e.g. MaxPoolImpl), you must add explicit template instantiation in the module’s .cpp file (e.g. search for template class in torch/csrc/api/src/nn/modules/pooling.cpp to see how this is done).
  • When you are adding a new C++ torch::nn::functional function:
    • How do we decide which file to put the new C++ torch::nn::functional function? Answer: We should look at where the corresponding torch::nn module is located. For example, torch::nn::PairwiseDistance lives in torch/csrc/api/include/torch/nn/modules/distance.h (which, following the rule above, is determined by torch/nn/modules/distance.py), so torch::nn::functional::pairwise_distance should live in torch/csrc/api/include/torch/nn/functional/distance.h.
    • You must add tests for the new functional in test/cpp/api/functional.cpp.
    • If you add a new header file for the functional, you must add include statement for the header file in torch/csrc/api/include/torch/nn/functional.h.
    • The name of the functional must match the Python version.
  • When you are adding a new module/functional options:
    • If the functional options only has optional arguments, the functional's function signature should have options = {}, to allow users to call the functional without passing options. And we must add a test for this.
    • If the functional / module options only has one non-optional argument, we need the implicit constructor for the options that only takes the non-optional argument (i.e. /* implicit */ Options(value_type value);), so that we are able to do functional(input, options_arg_value) / auto m = Module(options_arg_value) instead of functional(input, Options(options_arg_value)) / auto m = Module(Options(options_arg_value)). And we must add a test for this. (See DropoutOptions in torch/csrc/api/include/torch/nn/options/dropout.h as example.)
    • for optional Tensor argument, we should use torch::Tensor (with Tensor() as default “null" value), and don’t use c10::optional<Tensor>.
    • If the options is templatized (e.g. MaxPoolOptions in torch/csrc/api/include/torch/nn/options/pooling.h), you must add explicit template instantiation in the options’ .cpp file (search for template struct in torch/csrc/api/src/nn/options/pooling.cpp to see how this is done).
  • Do not use torch::IntArrayRef, use std::vector<int64_t> instead.

How do I run tests?

Please ask for @yf225's review when you open a PR to add a torch::nn module from this list.

Tracking post on PyTorch forum: https://discuss.pytorch.org/t/55650

cc @yf225

Metadata

Metadata

Assignees

Labels

good first issuemodule: cppRelated to C++ APImodule: nnRelated to torch.nntriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions