Customize stopword for Lucene Index #9708
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jasperjiaguo
force-pushed
the
lucene_text
branch
from
b5e3536 to
0cb9609
Compare
Codecov Report
@@ Coverage Diff @@
## master #9708 +/- ##
============================================
- Coverage 69.91% 60.35% -9.56%
- Complexity 4855 5192 +337
============================================
Files 1950 1938 -12
Lines 104339 104020 -319
Branches 15804 15768 -36
============================================
- Hits 72945 62778 -10167
- Misses 26267 36550 +10283
+ Partials 5127 4692 -435
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java
Outdated
Show resolved
Hide resolved
jasperjiaguo
force-pushed
the
lucene_text
branch
from
4703975 to
49d0289
Compare
...segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
Outdated
Show resolved
Hide resolved
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java
Outdated
Show resolved
Hide resolved
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java
Outdated
Show resolved
Hide resolved
...segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
Outdated
Show resolved
Hide resolved
...segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
Outdated
Show resolved
Hide resolved
siddharthteotia
approved these changes
Nov 2, 2022
Contributor
siddharthteotia
left a comment
siddharthteotia
left a comment
There was a problem hiding this comment.
LGTM overall. Mostly minor comments.
somandal
approved these changes
Nov 2, 2022
...java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java
Outdated
Show resolved
Hide resolved
jackjlli
approved these changes
Nov 2, 2022
...in/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java
Outdated
Show resolved
Hide resolved
Member
There was a problem hiding this comment.
What if stopWordsInclude and stopWordsExclude have the same word, will the code throw any exception on that?
Contributor
Author
There was a problem hiding this comment.
The current behavior is to prioritize exclude these duplicate words. I'm planning to put default ENGLISH_STOP_WORDS_SET, config keys, and this behavior in the user doc.
jackjlli
reviewed
Nov 2, 2022
...segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
Outdated
Show resolved
Hide resolved
amrishlal
reviewed
Nov 2, 2022
...al/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
Outdated
Show resolved
Hide resolved
Contributor
|
@jasperjiaguo can you also rebase once if not already. Just in case we see some spurious failures again. |
jasperjiaguo
force-pushed
the
lucene_text
branch
from
03572dd to
5c7c949
Compare
Contributor
|
The tests pass locally and this failed even after retriggering couple of times. Merging this |
Jackie-Jiang
added
feature
Configuration
Contributor
|
Thanks for adding the feature. Can you please update the PR description with an example on how to configure the stop words? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses a production scenario where we see
IT(Information Technology) in text data but it was pruned out from the text index asitis in theENGLISH_STOP_WORDS_SET. Therefore, we would like to make the stop word set configurable per column.The way to use this is to add
"stopWordInclude":"incl1, incl2, ..."and"stopWordExclude":"IT, excl2, excl3 ..."in thepropertiesof field config. As will include/exclude words from the default stop words list of"a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "than", "there", "these", "they", "this", "to", "was", "will", "with", "those"