Codecov Report

Merging #1881 into master will increase coverage by 1.35%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1881      +/-   ##
==========================================
+ Coverage   82.72%   84.08%   +1.35%     
==========================================
  Files          97       97              
  Lines       14316    14316              
==========================================
+ Hits        11843    12037     +194     
+ Misses       2473     2279     -194

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3386d9...821ba9e. Read the comment docs.

Hi, thanks for looking into it! What's your use-case for adding 207 special tokens?

In the kaggle Tensorflow 2 Nature question competition. I try to add some additional Sequence Embedding Such as [Tableid=13] and split short sentence.

I may misunderstand, but why not use the add_tokens method rather than the add_special_tokens method, which is reserved for tokens like CLS or MASK?

Yes, add_special_tokens method is reserved for a limited number of tokens with special properties and usage like CLS or MASK. For other uses, go for add_tokens.

Here is how we solved the performance issue when adding custom vocabulary: In the add_tokens method, we simply integrate new_tokens into the self.vocab.

from transformers import BertTokenizer, WordpieceTokenizer
from collections import OrderedDict


class CustomVocabBertTokenizer(BertTokenizer):
    def add_tokens(self, new_tokens):
        new_tokens = [token for token in tokens if not (token in self.vocab or token in self.all_special_tokens)]

        self.vocab = OrderedDict([
            *self.vocab.items(),
            *[
                (token, i + len(self.vocab))
                for i, token in enumerate(new_tokens)
            ]
        ])

        self.ids_to_tokens = OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab, unk_token=self.unk_token)

        return len(new_tokens)