System Info

I get the following error when running transformers-cli env:

Traceback (most recent call last):
  File "/Users/micah/opt/anaconda3/envs/fBERT/bin/transformers-cli", line 7, in <module>
    from transformers.commands.transformers_cli import main
  File "/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/transformers/commands/transformers_cli.py", line 26, in <module>
    from .user import UserCommands
  File "/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/transformers/commands/user.py", line 20, in <module>
    from huggingface_hub.hf_api import HfFolder, create_repo, list_repos_objs, login, logout, whoami
ImportError: cannot import name 'list_repos_objs' from 'huggingface_hub.hf_api' (/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/huggingface_hub/hf_api.py)

4.16.2 is the Transformers library version I'm using. I don't think any other info is relevant.

Who can help?

Copying from the GPT2 list above: @patil-suraj, @patrickvonplaten, @LysandreJik

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I encountered a pretty ridiculous failure mode (with my own training pipeline, and no modification to the transformers library) in which, using GPT2 (imported as from transformers import GPT2Config, GPT2Model):

• I was getting almost 0 training and validation loss
• I was getting very bad performance when feeding the model incomplete sequences (e.g. for test-time word-generation)

After much debugging, I found that the issue was that the value of self.masked_bias (currently set to -10e4) is too low, which is supposed to implement the "mask" of the causal masked attention.

For some high enough learning rates (which are still stable, however), the network is able to find a hack to copy the input to the output (getting around the causal masking): just drive the attention weights lower than -10e-4, and the causal mask will effectively not be doing anything! This means that the model will be able to attend the whole input in trying to generate the output (even the future tokens), so it will be able to simply copy it at training and validation time (getting 0 loss).

What I found while debugging this line (before the softmax call):

>>> attn_weights[0,0,:4,:4]
[[-15258.9805, -10000.0000, -10000.0000, -10000.0000],
        [-15044.7910, -16940.4766, -10000.0000, -10000.0000],
        [-11722.1553, -13301.4287,  -1438.0649, -10000.0000],
        [ -9711.6445, -11315.6006,  -1065.3066, -12052.6035]]

As one can see, the attention weights outside of the causal masking are even larger than this fixed value, which lead the softmax to return non-zero weights on all inputs. The off-upper-triangle entries should be much larger, so that the causal-mask-triangle entries get assigned 0 weight.

Expected behavior

In terms of fixes, there should probably be an assertion error that the weights never go below self.masked_bias, or setting it to an even lower value. Alternatively, it could return NaNs? It should definitely not fail silently.