🚀 The feature, motivation and pitch
Recent work in transformer architectures such as T5 [1] and Attention with Linear Biases (ALiBi) [2] have shown that moving the positional encodings from the word embedding layer and directly into the self-attention computation improves the capacity of models to extrapolate to longer sentences while keeping lower perplexity scores. Both T5 and ALiBi add biases to the logits resulting from the QK multiplication, turning the softmax operation into something like this:
weights = softmax(QK^T + B), where B can either be a constant pre-defined matrix, like in ALiBi, or a learnable bias, like in T5. In both cases, the values of B are dependent on the relative distance between tokens, with the farthest tokens usually having the highest penalties.
Usually, this is implemented through modification of the attention mask to add these extra weights on top, changing it from a boolean yes/no attention to a weighted kind of model.
Performance-wise, our experiments are summarized in the graph below, with @daviswer explaining: "Here we train 3 AliBi decoders (390M params) with different sequence lengths (2048, 1024, 512) but perform validation at sequence length 2048 all the time. This allows us to track the degree of overfitting to sequence length over time. We then do the same for 3 equivalent models using T5 relative positional bias (rpb) instead of AliBi. While val perplexity of both variants suffers greatly over time with diverging train/val sequence lengths, the degree of increase is worse for rpb, indicating that while generalization to longer sequence lengths is a difficult problem, AliBi is indeed better at it than rpb." From the ALiBi paper we also know that both perform better than the traditional word embedding-level positional encodings.
As of right now, the BetterTransformers path in PyTorch only supports two models for the attention mask: none at all, or the one used for causal LMs. We propose modifying the current implementation so that more complex masks like the ones required for relative positional biases and ALiBi can also be supported.
cc: @daviswer @supriyogit @raghukiran1224 @mudhakar @mayank31398 @cpuhrsch @HamidShojanazeri
[1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (https://arxiv.org/abs/1910.10683)
[2] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (https://arxiv.org/abs/2108.12409)
Alternatives
No response
Additional context
No response
cc @jbschlosser @bhosmer @cpuhrsch @erichan1
🚀 The feature, motivation and pitch
Recent work in transformer architectures such as T5 [1] and Attention with Linear Biases (ALiBi) [2] have shown that moving the positional encodings from the word embedding layer and directly into the self-attention computation improves the capacity of models to extrapolate to longer sentences while keeping lower perplexity scores. Both T5 and ALiBi add biases to the logits resulting from the QK multiplication, turning the softmax operation into something like this:
weights = softmax(QK^T + B), where B can either be a constant pre-defined matrix, like in ALiBi, or a learnable bias, like in T5. In both cases, the values of B are dependent on the relative distance between tokens, with the farthest tokens usually having the highest penalties.
Usually, this is implemented through modification of the attention mask to add these extra weights on top, changing it from a boolean yes/no attention to a weighted kind of model.
Performance-wise, our experiments are summarized in the graph below, with @daviswer explaining: "Here we train 3 AliBi decoders (390M params) with different sequence lengths (2048, 1024, 512) but perform validation at sequence length 2048 all the time. This allows us to track the degree of overfitting to sequence length over time. We then do the same for 3 equivalent models using T5 relative positional bias (rpb) instead of AliBi. While val perplexity of both variants suffers greatly over time with diverging train/val sequence lengths, the degree of increase is worse for rpb, indicating that while generalization to longer sequence lengths is a difficult problem, AliBi is indeed better at it than rpb." From the ALiBi paper we also know that both perform better than the traditional word embedding-level positional encodings.
As of right now, the BetterTransformers path in PyTorch only supports two models for the attention mask: none at all, or the one used for causal LMs. We propose modifying the current implementation so that more complex masks like the ones required for relative positional biases and ALiBi can also be supported.
cc: @daviswer @supriyogit @raghukiran1224 @mudhakar @mayank31398 @cpuhrsch @HamidShojanazeri
[1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (https://arxiv.org/abs/1910.10683)
[2] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (https://arxiv.org/abs/2108.12409)
Alternatives
No response
Additional context
No response
cc @jbschlosser @bhosmer @cpuhrsch @erichan1