System Info
transformers version: 4.33.2- Platform: Linux-6.2.0-33-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
Who can help?
@ArthurZucker
Information
Tasks
Reproduction
Hi,
When adding a token to the T5 tokenizer and then tokenizing a string, it seems that the encoding step is inserting an unwanted space after the added token.
from transformers import AddedToken, T5TokenizerFast
tokenizer = T5TokenizerFast.from_pretrained("google/flan-t5-small")
tokenizer.add_tokens(["<"])
print(tokenizer.encode("<body>")) # [32100, 643, 3155, 1]
print(tokenizer.decode(tokenizer.encode("<body>"))) # < body></s>
print(tokenizer.convert_ids_to_tokens(tokenizer.encode("<body>"))) # ['<', '▁body', '>', '</s>']It's unclear why the model is using the token "▁body" when "body" is also in the vocabulary? And even if "body" weren't in the vocabulary, I'd still expect convert_ids_to_tokens to give back something like ["<", "b", "o", "d", "y", ">", "</s>"].
Expected behavior
The following script should print <body></s>.
from transformers import AddedToken, T5TokenizerFast
tokenizer = T5TokenizerFast.from_pretrained("google/flan-t5-small")
tokenizer.add_tokens(["<"])
print(tokenizer.decode(tokenizer.encode("<body>")))I saw #24565 but this doesn't seem to have solved it for this case?
System Info
transformersversion: 4.33.2Who can help?
@ArthurZucker
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Hi,
When adding a token to the T5 tokenizer and then tokenizing a string, it seems that the encoding step is inserting an unwanted space after the added token.
It's unclear why the model is using the token
"▁body"when"body"is also in the vocabulary? And even if"body"weren't in the vocabulary, I'd still expectconvert_ids_to_tokensto give back something like["<", "b", "o", "d", "y", ">", "</s>"].Expected behavior
The following script should print
<body></s>.I saw #24565 but this doesn't seem to have solved it for this case?