This confused me a lot as I was trying to add common tests for Marian.
I believe special_tokens_mask must return 0 for positions with unk_token_id, but <unk> is in SpecialTokensMixin.all_special_ids.
Relatedly, this RobertaTokenizer code suggests there should be a relationship between all_special_ids and special_token_mask logic.
Maybe <unk> shouldnt be in SpecialTokensMixin.all_special_ids?
This confused me a lot as I was trying to add common tests for Marian.
I believe special_tokens_mask must return 0 for positions with unk_token_id, but
<unk>is inSpecialTokensMixin.all_special_ids.Relatedly, this RobertaTokenizer code suggests there should be a relationship between
all_special_idsandspecial_token_masklogic.Maybe
<unk>shouldnt be inSpecialTokensMixin.all_special_ids?