🐛 Bug

Information

I am using Bert-Base-cased model to train my custom Named entity recognition(NER) model with a sequence length of 512.

Language I am using the model on: English

The problem arises when using:

• the official example scripts: token-classification/run_ner.py

The tasks I am working on is:

• an official GLUE/SQUaD task: Named entity recognition
• my own task or dataset: Custom Dataset

To reproduce

Steps to reproduce the behavior:

1.Use the default NER Pipeline to load the custom trained model
self.model_prediction_pipeline = pipeline( "ner", model=model_path, tokenizer= model_path, grouped_entities=True )
2. I've attached the Evaluation results of the model.
eval_loss = 0.021479165139844086
eval_precision = 0.8725970149253731
eval_recall = 0.8868932038834951
eval_f1 = 0.8796870297923562
epoch = 5.0

Expected behavior

1. Model should produce a good accuracy corresponding to the F1 score.
2. However during Inference, I am not getting an accuracy over 30%
3. Not sure if the inconsistent tokenisation leads to poor results.

Environment info

• transformers version: 3.0.0
• Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.7
• PyTorch version (GPU?): 1.4.0
• Tensorflow version (GPU?):NA
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No