Skip to content

[LlamaTokenizerFast] Refactor default llama#28881

Merged
ArthurZucker merged 28 commits into
mainfrom
refactor-default-llama
Apr 23, 2024
Merged

[LlamaTokenizerFast] Refactor default llama#28881
ArthurZucker merged 28 commits into
mainfrom
refactor-default-llama

Conversation

@ArthurZucker

@ArthurZucker ArthurZucker commented Feb 6, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What does this PR do?

from transformers import LlamaTokenizerFast, AddedToken
tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", legacy=False, from_slow=True)
tokenizer.add_tokens([AddedToken("<REPR_END>", rstrip=True, lstrip=True)], special_tokens=False)
tokenizer.tokenize("<REPR_END>inform<s>. Hey.       .")
['<REPR_END>', 'in', 'form', '<s>', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']
tokenizer.tokenize("inform<s>. Hey.       .")
['in', 'form', '<s>', '.', '▁Hey', '.', '▁▁▁▁▁▁', '▁.']

This requires huggingface/tokenizers#1476 to be merged.
Finally FIXES ALL THE remaining tokenizer issues!

Fixes #29617 as well (will update Gemma one)
fixes #28577
fixes #29617
fixes #29626
fixes #29694
fixes #29868
fixes #29872
fixes #30416
enabled by huggingface/tokenizers#1476

@ArthurZucker ArthurZucker mentioned this pull request Feb 6, 2024
Closed
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Feb 6, 2024

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ArthurZucker
ArthurZucker commented Feb 20, 2024
View reviewed changes
Comment thread src/transformers/convert_slow_tokenizer.py
@ArthurZucker ArthurZucker mentioned this pull request Mar 7, 2024
Closed
@huggingface huggingface deleted a comment from github-actions Bot Mar 22, 2024
@ArthurZucker
ArthurZucker marked this pull request as ready for review March 22, 2024 09:07
This was referenced Mar 25, 2024
Closed
Closed
Closed
This was referenced Mar 25, 2024
Closed
Closed
Closed
Merged
@huggingface huggingface deleted a comment from github-actions Bot Apr 18, 2024
@ArthurZucker

ArthurZucker commented Apr 18, 2024

Copy link
Copy Markdown
Collaborator Author

Coming next release!

@ArthurZucker ArthurZucker mentioned this pull request Apr 23, 2024
Closed
4 tasks
LysandreJik
LysandreJik approved these changes Apr 23, 2024
View reviewed changes

@LysandreJik LysandreJik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds great to me

@vasqu vasqu mentioned this pull request Apr 23, 2024
Closed
4 tasks
pcuenca
pcuenca reviewed Apr 23, 2024
View reviewed changes
Comment thread src/transformers/models/llama/tokenization_llama_fast.py Outdated
ArthurZucker
ArthurZucker commented Apr 23, 2024
View reviewed changes
Comment thread src/transformers/convert_slow_tokenizer.py
Comment thread src/transformers/convert_slow_tokenizer.py Outdated
Comment thread src/transformers/convert_slow_tokenizer.py
)
user_defined_symbols = [
AddedToken(token, normalized=False, special=False) for token in proto.trainer_spec.user_defined_symbols
AddedToken(token, normalized=True, special=False) for token in proto.trainer_spec.user_defined_symbols

@ArthurZucker ArthurZucker Apr 23, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a must? Unless with only the split this works.

@ArthurZucker
ArthurZucker merged commit e34da3e into main Apr 23, 2024
@ArthurZucker
ArthurZucker deleted the refactor-default-llama branch April 23, 2024 21:13
This was referenced May 6, 2024
Closed
Closed
Closed
This was referenced Jun 5, 2024
Closed
Closed
@Alienmaster Alienmaster mentioned this pull request Jun 15, 2024
Closed
4 tasks
@hamishivi hamishivi mentioned this pull request Jul 23, 2024
Merged
@coderabbitai coderabbitai Bot mentioned this pull request Apr 12, 2026
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment