Just wondering why we need this helper specific for gelu? any idea for relu?

This is what we have in the test_nn.py.
I merged them into one test function for encoderlayer, but the decoder layer is contributed by external user who strictly followed the test_nn.py. I will refactor this a bit later

Should we publish transformer cpp later? The transformer model in general comes with decoder module, which I guess will be published later.

@zhangguanheng66 Current situation is that we have 2 people working on the transformer. Decoder will be in another PR. Pushing this first, so the decoder developer is easy to sync. And we will have Transformer top layer next week.

The second argument is called "mask" in Python:

I do prefer src_mask because it makes it clear what is getting masked...

@zhangguanheng66 I make the name to src_mask for the same reason as @zou3519, agree?

I think it makes sense to call src_mask and tgt_mask along with src_key_padding_mask and tgt_key_padding_mask (consistent with the python ones)

When does reset_parameters() get called? is it a user API or something that is used in the framework?

@zou3519 As I mentioned before, this is a convenient function to reset parameters within the module. It is not necessary to call it. And I believe, it won't be called in most of the case.

Hmm, so the thing I am confused about here is:

• there's no reset_parameters() or _reset_parameters on TransformerEncoderLayer in Python. Some of our Python nn.Modules do have reset_parameters() that re-initializes parameters.

And one potential concern is, let's say you have an optimizer that takes in a list of parameters (disclaimer: I have no clue how optimizers in C++ work). Is the following possible?

• I create a TransformerEncoderLayer in C++
• I add pass the parameters of a TransformerEncoderLayer to an optimizer (in C++)
• then, I call reset_parameters() on TransformerEncoderLayer
• The optimizer holds onto references of the original parameters (and not the new ones!)

This is not possible since the C++ module underlying is a shared pointer, and optimizer doesn't clone the module but just shallow copy the module, so this case should be ok.

But the reset_parameters() it self is tricky, I admit that, even in python impl, reset_parameters() has different usage in different modules and not always there and not always get called. There is a PR that trying to standarize it by making reset_parameter() virtual and moved that to base class, but this change doesn't make sense since half of the module doesn't have it and virtual will make extra cost. So currently, reset_parameters() is just an optional functions.

this also keeps a shallow copy of the data in encoder_layer, right?

@zou3519
No, TransformerEncoder will use encoder_layer_options as input, allocating a new TransformerEncdoerLayer.
This is the major difference between these two constructors.

Oh I see now, thanks for the clarification