common : fix common_chat_peg_parse for incomplete utf-8 sequence tail#19992
Open
akreal wants to merge 1 commit intoggml-org:masterfrom
Open
common : fix common_chat_peg_parse for incomplete utf-8 sequence tail#19992akreal wants to merge 1 commit intoggml-org:masterfrom
akreal wants to merge 1 commit intoggml-org:masterfrom
Conversation
akreal
force-pushed
the
strip_incomplete_utf8_tail
branch
from
2203f50 to
1f03175
Compare
Author
|
After more experiments it appeared that invalid UTF-8 sequence can appear also in the middle of string, so now the fix handles both mid-string and end-string conditions. But I am not sure if this is the correct place to do this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sometimes I experience an issue with
mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q8_0when it hits maximum generated tokens and leaves an incomplete UTF-8 sequence in the end of message (e.g. token IDs 1226 and 1156). This message then causescommon_chat_peg_parseto fail withFailed to parse input at pos 0error.Claude Sonnet 4.6 was used to analyze the problem and to generate the code. Everything was manually reviewed and tested.