Skip to content

add prefix space ignored in llama #29625#30964

Merged
itazap merged 4 commits into
mainfrom
add_prefix_space
May 24, 2024
Merged

add prefix space ignored in llama #29625#30964
itazap merged 4 commits into
mainfrom
add_prefix_space

Conversation

@itazap

@itazap itazap commented May 22, 2024

Copy link
Copy Markdown
Collaborator

What does this PR do?

  • add param add_prefix_space to llama tokenizer

Fixes #29625

Before submitting

  • modified test_no_prefix_space to test test_no_prefix_space=False

Who can review?

@ArthurZucker

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@itazap
itazap marked this pull request as ready for review May 22, 2024 14:03
@itazap
itazap requested a review from ArthurZucker May 22, 2024 14:03

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing! 🔥 Yep the test was missing the case when we actually do want it to be forcefully converted from slow

@itazap
itazap merged commit 7f6e874 into main May 24, 2024
@itazap
itazap deleted the add_prefix_space branch May 24, 2024 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add_prefix_space won't be respected by Llama tokenizer

3 participants