Environment info

• transformers version: 3.0.2
• Platform: Linux-4.15.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9

Who can help

tokenizers: @mfuntowicz

Information

Bert tokenizer treats [UNK] token as special during tokenizer.convert_ids_to_tokens(...,skip_special_tokens=True)
while tokenizer(...,return_special_tokens_mask=True) doesn't (for obvious reasons).
I think it would be better to preserve [UNK] tokens in convert_ids_to_tokens for consistency of the term "special token".

To reproduce

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
# fast tokenizer also has the same problem
# tokenizer = AutoTokenizer.from_pretrained("bert-large-cased",use_fast=True)
text="sentence 한 셔 word"
print('tokens:',tokenizer.tokenize(text))

tokens=tokenizer(text,return_special_tokens_mask=True)
print("input_ids:",tokens["input_ids"])
print("special_token_mask:",tokens["special_tokens_mask"])
no_special=tokenizer.convert_ids_to_tokens(tokens["input_ids"], skip_special_tokens=True)
special=tokenizer.convert_ids_to_tokens(tokens["input_ids"])
print('tokens from ids (skip special): ',no_special)
print('tokens from ids (skip special): ',special)

print('special tokens',tokenizer.all_special_tokens)

Expected behavior

Also I think keeping [UNK] would be a better behavior, as convert_ids_to_tokens is used in inference pipelines and using skip_special_token for getting rid of [CLS][SEP][PAD] tokens leads to unintended loss of [UNK] tokens, which is important.

Related: #4391