The docs recommend adding the special eos_token <\s> to the end of each string when encoding/decoding with T5Tokenizer. However, this (and the other special tokens e.g. unk_token, pad_token aren't assigned unique ids in the lookup vocabulary (they are mapped to {0,1,2}, which are indices for other common words in the vocab). In practice, I find my model fails to properly produce the eos_token since it is associated with blank spaces, so the model produces run-ons during generation

To reproduce

>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained('t5-base')
>>> tokenizer.pad_token
'<pad>'
>>> tokenizer.pad_token_id
0
>>> tokenizer.eos_token
'</s>'
>>> tokenizer.eos_token_id
1
>>> tokenizer.unk_token
'<unk>'
>>> tokenizer.unk_token_id
2

>>> tokenizer.decode([0])
''
>>> tokenizer.decode([1])
''
>>> tokenizer.decode([2])
' ⁇ '

Expected behavior

>>> tokenizer.decode([0])
'<pad>'
>>> tokenizer.decode([1])
'</s>'
>>> tokenizer.decode([2])
'<unk>'

Environment info

• transformers version: 2.9.1