-F -i misses files whose matches differ by a non-ASCII case fold when an index is used (CAFÉ, fullwidth ＩＤ, ſ/K) #173

Summary

With -F/--fixed-strings and case-insensitive matching (-i, or -S with an all-lowercase pattern), indexed searches silently drop files whose matches differ from the pattern by a non-ASCII case fold. This happens with the on-disk index and with tgrep serve. --no-index and ripgrep find these files, and so does the same pattern without -F.

It also misses the exact text: tgrep -l -F -i -- 'ＩＤ' . does not list a file that contains ＩＤ.

The search exits as if nothing matched and prints no warning.

Reproduce

mkdir repro && cd repro
printf 'CAF\303\211\n'                         > upper_accent.txt     # CAFÉ
printf '\357\274\251\357\274\244\n'            > upper_fullwidth.txt  # ＩＤ (U+FF29 U+FF24)
printf '\305\277hell\305\277hoc\342\204\252\n' > long_s_kelvin.txt    # ſhellſhocK (U+017F, U+212A)
printf 'caf\303\251 \357\275\211\357\275\204 shellshock\n' > lower.txt # café ｉｄ shellshock
tgrep index .

tgrep -l -F -i -- 'café' .
tgrep -l -F -i -- 'ｉｄ' .
tgrep -l -F -i -- 'shellshock' .
tgrep -l -F -S -- 'café' .
tgrep -l -F -i -- 'ＩＤ' .

• Expected: the same files as tgrep --no-index and ripgrep.
• Actual: files whose matching text contains a non-ASCII case variant are missing.

--stats shows the fixed-string plan requiring trigrams the uppercase file does not have, while the regex plan falls back safely:

$ tgrep -l --stats -F -i -- 'ｉｄ' .
./lower.txt
Query plan: AND(4 trigrams) (candidates: 1/4; raw candidates: 1/4)

$ tgrep -l --stats -i -- 'ｉｄ' .
./upper_fullwidth.txt
./lower.txt
Query plan: MatchAll (full scan) (candidates: 4/4; raw candidates: 4/4)

Environment

• tgrep 1.0.11, built from ad8fffa (current main) with rustc 1.95.0 (59807616e 2026-04-14)
• Windows 11 Pro (build 28000); the code involved is platform-independent
• The café and ｉｄ cases fail the same way through tgrep serve

Cause

build_multi_pattern_plan sends fixed-string patterns to build_literal_plan (query.rs#L78-L79). It lowercases the whole literal with Unicode str::to_lowercase and requires every trigram of the result:

The index stores only trigrams of the original text and of its ASCII-lowercased form (extract_merged_masks). So:

• CAFÉ is indexed with É (C3 89), but the plan requires é (C3 A9).
• ＩＤ is never folded in the index, but the plan requires ｉｄ.
• The matcher folds ſ (U+017F) and K (U+212A, Kelvin sign) to s and k, but the plan requires ASCII bytes that the file does not contain.

The regex path has handled this since #163. ascii_class_literal merges only single-ASCII case classes into a literal run and breaks the run on classes with non-ASCII members. inline_case_flags_narrow_candidates_without_losing_matches covers ſ, K and É. The fixed-string path was not updated, and test_case_insensitive_literal_plan asserts the fully lowercased trigrams.

The CLI (search.rs#L1291) and the server (serve.rs#L1703) both build plans through build_multi_pattern_plan, so both are affected.

Possible fix

When case_insensitive is set, plan the escaped literal with the regex planner, for example build_query_plan(&regex_syntax::escape(literal), true). That makes -F -i as safe as -i. The cost is the selectivity the regex path already gives up: s and k fold to non-ASCII code points, so they break literal runs.

If that cost matters, build_literal_plan could keep ASCII letters in the run and break it only at non-ASCII characters that have other case forms. That fixes the É and ＩＤ cases without the selectivity loss, but still misses ſ/K equivalents, as the fixed-string path does today.

A regression test could index the four files above and check that -F -i returns the same files as --no-index.