You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4501552
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](inverted-index) Preserve effective IK identities and index bindings
### What problem does this PR solve?
Related PR: apache#67918
Problem Summary:
Explicit built-in IK requests could select a smart or differently configured
physical index after distinct IK identities were admitted on one column.
Named A-to-a character replacement was not folded through IK's default
lowercasing, and word delimiter concatenation discarded source gaps needed
by downstream pinyin offsets. Encoded selection keys could also collide
with arbitrary legal policy names.
Match built-in IK indexes by effective configuration and carry the existing
mode, lowercase and outer character-filter fields into BE selection keys.
Separate encoded configurations from policy names without changing provider
bindings, Thrift or stored index formats. Canonicalize named character
filters in tokenizer context, and preserve optional per-rune source ends
through concatenation and reset. Run the two new global-policy suites in
the nonConcurrent group to avoid contributing to concurrent quota exhaustion.
### Release note
Fix IK MATCH index selection and effective analyzer duplicate detection,
and preserve pinyin source offsets through word delimiter concatenation.
### Check List (For Author)
- Test
- [x] Regression test
Official runner: test_analyzer_identity_semantics and
test_ik_custom_analyzer passed together; test_ik_analyzer passed
separately. Expected output was generated by the runner and then
checked in a normal run on a task-owned ASAN cluster.
- [x] Unit Test
Official ASAN BE runner: 178 tests across 12 related suites passed.
After clang-tidy fixes, all 120 tests in the three affected suites
passed again. Official FE runner: 118 tests in four classes passed.
The three review defects and the encoded-name collision have
discriminating native RED/GREEN evidence.
- [ ] Manual test
- [ ] No need to test or manual test. Explain why:
- Additional validation
Native build.sh --be --fe passed; BE was rebuilt after the final source
change. Java Checkstyle and build hygiene passed. Clang-tidy checked all
12 newly changed C++ files and rechecked the three files subsequently
corrected, with no checks disabled. The run remains qualified by
pre-existing processCurrentToken size/complexity diagnostics and
diagnostics in unchanged CLucene and Boost headers. A baseline-only
unmatched NOLINTEND comment was removed through a temporary VFS overlay,
not by changing repository code. A controlled baseline source comparison
confirmed the function's complexity decreased from 75 to 69 and its
length from 141 to 125 lines. These are not claimed as a clean tidy pass.
- Behavior changed:
- [ ] No.
- [x] Yes. Explicit built-in IK uses its matching default configuration;
distinct physical configurations retain separate selection keys,
equivalent named character filters are rejected as duplicates, and
concatenated tokens retain their original source spans.
- Does this need documentation?
- [x] No. This corrects existing analyzer behavior and test isolation.
- [ ] Yes.
0 commit comments