Repository navigation
Advance past the whole codepoint after a zero-width regex match - #3586
Merged
Merged
Conversation
After a zero-width match in a global ("g") regex, f_match advanced the
search start by one byte to avoid re-matching at the same spot. But
Oniguruma runs in UTF-8, so on a multibyte character that +1 byte lands
inside the next codepoint, producing a spurious extra zero-width match at
a duplicated offset. So `"aé" | [match("";"g")|.offset]` gave [0,1,2,2]
instead of [0,1,2], and `"aé" | gsub("";"-")` gave "-a-é--" instead of
"-a-é-"; match, gsub, sub, splits, split/2 and scan were all affected on
multibyte input.
Advance by the codepoint length (jvp_utf8_decode_length, already used in
this function) instead of a single byte. This completes the ASCII-only
fix from jqlang#2565 for the multibyte case; the existing ASCII zero-width
tests are unchanged.
Fixes jqlang#3585
Member
|
Nice 👍 Huh guess i ran into this and somehow didn't realize it was a bug while working on #3432. I noticed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
After a zero-width match in a global (
"g") regex,f_matchadvanced the search start by one byte to avoid re-matching at the same position. Oniguruma runs inONIG_ENCODING_UTF8, so on a multibyte character that+1byte lands inside the next codepoint, and the nextonig_searchfinds a spurious extra zero-width match at a duplicated offset — one bogus match per multibyte character.match,gsub,sub,splits,split/2andscanwere all affected on multibyte input:Fix
Advance past the whole codepoint using
jvp_utf8_decode_length(already used elsewhere inf_match) rather than a single byte, so the search start can never land inside a multibyte character. This is the multibyte tail of #2565 — the ASCII case it fixed (and its test) still pass.Tests
Added multibyte regression cases to
tests/onig.test(offsets over 2/3/4-byte chars incl.€/🚀,gsub,splits). They fail on the previous code and pass now../jq --run-tests tests/onig.test→ 50/50; fullmake checkgreen.Fixes #3585