🐛 Bug

Information

Model I am using (Bert, XLNet ...): 'ner' pipeline

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Have transformers 3.0.2 installed
2. Run the below code

from transformers import pipeline
nlp = pipeline('ner', grouped_entities=True)
nlp('Welcome to New York')

Expected behavior

We should receive [{'entity_group': 'I-LOC', 'score': 0.9984402656555176, 'word': 'New York'}, but instead the output has duplicated 'New York': [{'entity_group': 'I-LOC', 'score': 0.9984402656555176, 'word': 'New York'}, {'entity_group': 'I-LOC', 'score': 0.9984402656555176, 'word': 'New York'}].

The Cause of the Issue According to Me

After reading 3.0.2, I noticed that lines 1047-1049 were added. I think this was done to fix a prior issue that caused the last named entity in the sequence to be occasionally omitted when grouped_entities=True. Long story short, I think this snippet was a patch that only shifted the problem from being an occasional named entity omission to an occasional named entity duplicate.

The for-loop that precedes this snippet is inconsistent in that sometimes the last named entity gets successfully added anyway (e.g. if the if clause on 1025 (first iteration) or 1032 is entered on the last iteration). In this case, there is a duplicate entry upon the calling of the new code at 1047. On the converse, the last named entity won’t be added if the else clause in line 1041 is entered on the last iteration. In this case, the final named entity correctly gets added after the new code snippet is run.

In short, there is a duplicate (I think) if (i) there is only one recognized named entity or (ii) the last named entity is one such that the tokenizer cut it up into multiple tokens. Otherwise, there is no duplicate.

nlp(‘Welcome to Dallas’) -> duplicate 'Dallas' because 'Dallas' is the only named entity
nlp(‘HuggingFace is not located in Dallas’) -> no duplicate because there are multiple entities and the final one 'Dallas' is not tokenized into multiple tokens
nlp(‘HuggingFace is located in New York City’) -> duplicate ‘New York City’ because the final named entity 'New York City' is tokenized into multiple tokens

Environment info

• transformers version: 3.0.2
• Platform: Linux-5.3.0-1031-azure-x86_64-with-glibc2.10
• Python version: 3.8.1
• PyTorch version (GPU?): 1.5.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no