🚀 Feature request
Good Second Issue - A more advanced issue for contributors who want to dive more into Longformer's attention mechanism.
Longformer currently only outputs global attentions, which is suboptimal because users might be interested in the local attentions as well. I propose to change the "output_attention" logic as follows in longformer:
attentions should correspond to the "local" attentions and then we'll add a new output type global_attention that contains the global_attentions. This is consistent with the naming of attention_mask and global_attention_mask IMO and the cleanest way to implement the feature.
Implementing this feature would mean to that Longformer will require its own ModelOutput class =>
BaseModelOutput, => LongformerBaseModelOutput or BaseModelOutputWithGlobalAttention (prefer the first name though)
BaseModelOutputWithPooling, => ...
Also some tests will have to be adapted.
This is a slightly more difficult issue, so I'm happy to help on it. One should understand the difference between local and global attention and how Longformer's attention is different to e.g. Bert's attention in general.
For more detail check out discussion here: #5646
🚀 Feature request
Good Second Issue - A more advanced issue for contributors who want to dive more into Longformer's attention mechanism.
Longformer currently only outputs global attentions, which is suboptimal because users might be interested in the local attentions as well. I propose to change the "output_attention" logic as follows in longformer:
attentionsshould correspond to the "local" attentions and then we'll add a new output typeglobal_attentionthat contains the global_attentions. This is consistent with the naming ofattention_maskandglobal_attention_maskIMO and the cleanest way to implement the feature.Implementing this feature would mean to that Longformer will require its own
ModelOutputclass =>BaseModelOutput,=>LongformerBaseModelOutputorBaseModelOutputWithGlobalAttention(prefer the first name though)BaseModelOutputWithPooling,=> ...Also some tests will have to be adapted.
This is a slightly more difficult issue, so I'm happy to help on it. One should understand the difference between local and global attention and how Longformer's attention is different to e.g. Bert's attention in general.
For more detail check out discussion here: #5646