Skip to content

Advance past the whole codepoint after a zero-width regex match - #3586

Merged
itchyny merged 1 commit into
jqlang:masterfrom
winklemad:fix-multibyte-zero-width-match
Jul 19, 2026
Merged

itchyny merged 1 commit into
jqlang:masterfrom
winklemad:fix-multibyte-zero-width-match

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

What / why

After a zero-width match in a global ("g") regex, f_match advanced the search start by one byte to avoid re-matching at the same position. Oniguruma runs in ONIG_ENCODING_UTF8, so on a multibyte character that +1 byte lands inside the next codepoint, and the next onig_search finds a spurious extra zero-width match at a duplicated offset — one bogus match per multibyte character. match, gsub, sub, splits, split/2 and scan were all affected on multibyte input:

$ echo '"aé"'     | jq -c '[match("";"g")|.offset]'   # before [0,1,2,2]   after [0,1,2]
$ echo '"héllo"'  | jq -c 'gsub("(?=.)";"-")'         # before "-h-é--l-l-o"   after "-h-é-l-l-o"
$ echo '"aé"'     | jq -c '[splits("")]'              # before ["","a","é","",""]   after ["","a","é",""]

Fix

Advance past the whole codepoint using jvp_utf8_decode_length (already used elsewhere in f_match) rather than a single byte, so the search start can never land inside a multibyte character. This is the multibyte tail of #2565 — the ASCII case it fixed (and its test) still pass.

Tests

Added multibyte regression cases to tests/onig.test (offsets over 2/3/4-byte chars incl. €/🚀, gsub, splits). They fail on the previous code and pass now. ./jq --run-tests tests/onig.test → 50/50; full make check green.

Fixes #3585

After a zero-width match in a global ("g") regex, f_match advanced the
search start by one byte to avoid re-matching at the same spot. But
Oniguruma runs in UTF-8, so on a multibyte character that +1 byte lands
inside the next codepoint, producing a spurious extra zero-width match at
a duplicated offset. So `"aé" | [match("";"g")|.offset]` gave [0,1,2,2]
instead of [0,1,2], and `"aé" | gsub("";"-")` gave "-a-é--" instead of
"-a-é-"; match, gsub, sub, splits, split/2 and scan were all affected on
multibyte input.

Advance by the codepoint length (jvp_utf8_decode_length, already used in
this function) instead of a single byte. This completes the ASCII-only
fix from jqlang#2565 for the multibyte case; the existing ASCII zero-width
tests are unchanged.

Fixes jqlang#3585

@itchyny itchyny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@itchyny
itchyny merged commit f1f39b4 into jqlang:master Jul 19, 2026
29 checks passed
@wader

wader commented Jul 19, 2026 •

Copy link
Copy Markdown
Member

Nice 👍 Huh guess i ran into this and somehow didn't realize it was a bug while working on #3432. I noticed jq -n '["🚀" | match(""; "g")] | length' behaved differently. BTW i rebased the pcre2 PR and all test pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zero-width regex match emits duplicate offsets on multibyte input (match/gsub/splits/scan)

3 participants