System Info
ubuntu
transformers==4.39.0dev
Who can help?
No response
Information
Tasks
Reproduction
Refer to tokenization test:
|
special_token = tokenizer.all_special_tokens[0] |
|
|
|
text = special_token + " aaaaa bbbbbb low cccccccccdddddddd l " + special_token |
|
text2 = special_token + " AAAAA BBBBBB low CCCCCCCCCDDDDDDDD l " + special_token |
|
|
|
toks_before_adding = tokenizer.tokenize(text) # toks before adding new_toks |
|
|
|
new_toks = ["aaaaa bbbbbb", "cccccccccdddddddd", "AAAAA BBBBBB", "CCCCCCCCCDDDDDDDD"] |
|
added = tokenizer.add_tokens([AddedToken(tok, lstrip=True, rstrip=True) for tok in new_toks]) |
|
self.assertIn(added, [2, 4]) |
|
|
|
toks_after_adding = tokenizer.tokenize(text) |
|
toks_after_adding2 = tokenizer.tokenize(text2) |
|
|
|
self.assertEqual(len(toks_after_adding), len(toks_after_adding2)) # Length should still be the same |
|
self.assertNotEqual( |
|
toks_after_adding[1], toks_after_adding2[1] |
|
) # But at least the first non-special tokens should differ |
from transformers import AutoTokenizer
from transformers.tokenization_utils import AddedToken
tokenizer = AutoTokenizer.from_pretrained("/path/to/llama2", use_fast=False)
special_token = tokenizer.all_special_tokens[0]
text = special_token + " aaaaa bbbbbb low cccccccccdddddddd l " + special_token
text2 = special_token + " AAAAA BBBBBB low CCCCCCCCCDDDDDDDD l " + special_token
toks_before_adding = tokenizer.tokenize(text) # toks before adding new_toks
new_toks = ["aaaaa bbbbbb", "cccccccccdddddddd", "AAAAA BBBBBB", "CCCCCCCCCDDDDDDDD"]
added = tokenizer.add_tokens([AddedToken(tok, lstrip=True, rstrip=True) for tok in new_toks])
toks_after_adding = tokenizer.tokenize(text)
toks_after_adding2 = tokenizer.tokenize(text2)
print(toks_after_adding, toks_after_adding2)If use_fast=False, the output is:
['<s>', 'aaaaa bbbbbb', '▁low', 'cccccccccdddddddd', '▁l', '▁', '<s>'] ['<s>', 'AAAAA BBBBBB', '▁low', 'CCCCCCCCCDDDDDDDD', '▁l', '▁', '<s>']
Otherwise
['<s>', '▁', '▁aaaaa▁bbbbbb', '▁low', '▁cccccccccdddddddd', '▁l', '▁', '<s>'] ['<s>', '▁', '▁AAAAA▁BBBBBB', '▁low', '▁CCCCCCCCCDDDDDDDD', '▁l', '▁', '<s>']
Expected behavior
Seems that lstrip=True did not take effect. When use_fast=True, the output should be the same as use_fast=False
System Info
ubuntu
transformers==4.39.0dev
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Refer to tokenization test:
transformers/tests/test_tokenization_common.py
Lines 864 to 881 in 9acce7d
If
use_fast=False, the output is:Otherwise
Expected behavior
Seems that
lstrip=Truedid not take effect. Whenuse_fast=True, the output should be the same asuse_fast=False