Environment info

• transformers version: 4.8.2
• Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.7.10
• PyTorch version (GPU?): 1.9.0+cu102 (True)
• Tensorflow version (GPU?): 2.5.0 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

• tokenizers: @LysandreJik (Specifically the RoBERTa / GPT tokenizer @patrickvonplaten)

Information

Model I am using is RoBERTa.

The problem arises when using:

• my own modified scripts:

A simple script that uses RoBERTa to do NER.

The tasks I am working on is:

• my own task or dataset:

I am doing Named Entity Recognition (NER) on the conll2003 dataset from the datasets library.

As such, I am using RoBERTa + a classification head on top to classify each token in the sequence.

Moreover, when the RoBERTa Tokenizer splits a word into many sub-tokens, I pass the entire sentence through RoBERTa then, using the word_ids returned by Tokenizer.batch_encode_plus, pass only the contextual embeddings associated with the first sub-token of each word into my final classification head. (otherwise, the len(prediction) > len(label)).

Detailed code of this can be found in the final Section below.

The Problem

The problem is with the word_ids() returned by batch_encode_plus() for sentences that have alphanumeric tokens like '18th' or '1980s'. Where the word_ids() will be as follows:

['During', 'Ġthe', 'Ġ1980', 's', 'Ġ,', 'Ġlife', 'Ġwas', 'Ġweird'] # No 'Ġ' before 's', as expected, but
word_ids = [None, 0, 1, 2, 3, 4, 5, 6, 7, None] # This causes a problem ! I expect it to be 
word_ids = [None, 0, 1, 2, 2....

['An', 'Ġ18', 'th', 'Ġcentury', 'Ġpoet'] # No 'Ġ' before 'th', as expected, but
word_ids = [None, 0, 1, 2, 3, 4, None, None, None, None] # This causes a problem ! I expect it to be 
word_ids = [None, 0, 1, 1....

Notice that the token '1980s' was split into ['Ġ1980', 's'] but the word_ids did NOT indicate this, as what is returned is [None, 0, 1, 2, 3, 4, 5, 6, 7, None]. Which indicates that the sub-token 's' is its own word (and NOT a sub-token of the word '1980s')

To reproduce

Steps to reproduce the behavior:

1. Import and Initialize the RoBERTa Tokenizer (Fast)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True)

2. batch_encode_plus sentences that have alphanumeric tokens like '18th' and '1980s':

sentences = ["During the 1980s , life was something else", "An 18th century poet"]

e = tokenizer.batch_encode_plus(sentences, return_tensors='pt', padding=True)

3. Print and inspect the word_ids(i)

print(tokenizer.tokenize(sentences[0]))
print(e.word_ids(0))

print(tokenizer.tokenize(sentences[1]))
print(e.word_ids(1))

Expected behavior

The word_ids should correctly indicate whenever tokens such as '1980s' and '18th' are split:

['<s>',   'An', 'Ġ18', 'th', 'Ġcentury', 'Ġpoet', '</s>']
[None,     0,      1,    1,        2,         3,   None]

Detailed Code

input_sentence = ["He lives joyfully"]
label          = ["O",  "O",   "O"]

tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
model = AutoModel.from_pretrained("roberta-base")

encoded_x = tokenizer.batch_encode_plus(input_sentence, return_tensors='pt', padding=True)
# The input sentence now becomes ["<s>", "ĠHe", "Ġlives", "Ġjoy", "fully", "</s>"]

contextual_embeddings = model(encoded_x.input_ids).last_hidden_state  # [1, 6, 768] tensor.

# I need to pass a [1, 3, 768] tensor into my final classification head
# So, I wrote a function that takes as input the word_ids
# and returns a list of the first sub-token of each word (dropping <s> and </s>)
# Function NOT included here for brevity. Same function works perfectly for BERT
my_function( [None, 0, 1, 2, 2, None] ) -> [0, 1, 2]

first_subtoken = torch.LongTensor([0, 1, 2])

embeddings_of_interest = contextual_embeddings[:, first_subtoken, :]  #  [1, 3, 768] tensor