Add full citation

added.

Is it intended behavior to have weight sharing? In a standard transformer (like from Attention is All You Need), I don't believe the weights are shared. I understand there has been followup work showing weight sharing can be beneficial in some cases, but if this is intended behavior, it might be useful to specify this difference from the paper's implementation in docs after the Vaswani et al. citation and cite the followup work.

As a broader point:
This seems to get at the fact that a Transformer is a rather high level component, without really consensus yet on its architecture with a lot decisions likely domain specific (weight sharing?, beam search decoder?, convolutions between layers?, output the intermediate attentions? hierarchy / grouping components of the task?, sparsity?... and many more ideas that are being rapidly published)

I don't mean to try and start needless debates, or to at all imply this Transformer code is not useful, but I will add in my thoughts that from my limited perspective that maybe Transformers might better belong in the contrib module or in the docs as a example that people can modify to meet their needs. The torch.nn.Modules module seems (at least currently) to be for more foundational components that can be composed into larger models, not full model architectures themselves.

There has already been fairly extensive discussion on this in #10459 , and it seemed like the consensus there was to focus on things like MultiHeadedAttention or PositionalEncoding for core and keep full architectures separate in the codebase.

As an addendum to that second part: Thinking about this more I could see how a TransformerEncoder (or like StackedSelfAttention or RecurrentSelfAttention) could maybe be considered a primitive component which could lend itself to maybe eventually being low level optimized and composed into novel things (though not really that much more primitive than something like a ResNet-block). However, when it starts seeming particularly high level, and full-archetecture-y is the inclusion of a prepackaged encoder and decoder Transformer for seq2seq, which, without a lot of additional components, seems likely would not meet all needs and could not be easily adapted/composed.
I don't know, and I don't really have any reason to hold strong opinions here. Feel free to dismiss this second part without real justification...

@DNGros Thanks for the comments. To your first question, it's not supposed to share the weights between layers. Actually, if you take a quick look at _get_clones function, it calls a "deepcopy" function.

For your second comment, we do realize there are many ongoing discussions about the transformer models and several variants in different domains. We want to provide a baseline for the research community and startups when people don't want to code from scratch. This module could help people try some preliminary ideas for the fast delivery. For some advanced users, we expect them to develop a specific transformer model, and they could possibly use this module as a reference or benchmark case.

As you suggest, we try to make the model highly "modularized". People can use nn.Tranformer, nn.TransformerEncoder, or EVEN nn.TransformerEncoderLayer, as needed. There is actually a word language example (pytorch/examples#555) where we use nn.TransformerEncoder as the seq2seq model.

In the future, if we see a variant of the transformer model requested widely by the community, we will continuously implement them in our framework. I see a baseline model could benefit more users as we optimize the module performance in the future.

@zhangguanheng66 Ok, sorry. I missed that.

Thanks for clarifying!