Skip to content

Conversation

@goerch
Copy link
Contributor

@goerch goerch commented Aug 22, 2023

White space escaping was wrong. Added more tests.

@klosax klosax linked an issue Aug 22, 2023 that may be closed by this pull request
4 tasks
@ggerganov ggerganov merged commit 46ef5b5 into ggml-org:master Aug 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The new tokenizer no longer encode space properly

2 participants