Json extract index filter support #12683
Conversation
|
@wirybeaver @Jackie-Jiang Split the original PR (https://github.com/apache/pinot/pull/12532/files) into two as discussed. Please take a look at this one for filter support once #12532 is merged. Ty! |
…o jsonExtractIndexFilterSupport
…o jsonExtractIndexFilterSupport
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #12683 +/- ##
============================================
- Coverage 61.75% 61.53% -0.23%
+ Complexity 207 198 -9
============================================
Files 2436 2456 +20
Lines 133233 134439 +1206
Branches 20636 20817 +181
============================================
+ Hits 82274 82721 +447
- Misses 44911 45531 +620
- Partials 6048 6187 +139
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| return getMatchingFlattenedDocIds(filter, false); | ||
| } | ||
|
|
||
| private RoaringBitmap getMatchingFlattenedDocIds(FilterContext filter, boolean allowNestedExclusivePredicate) { |
There was a problem hiding this comment.
Trying to understand why we cannot always allow nested exclusive predicate?
| } | ||
| return matchingDocIds; | ||
| } | ||
| case PREDICATE: { |
There was a problem hiding this comment.
Should we add NOT since we allow exclusive predicate?
|
|
||
| @Override | ||
| public Map<String, RoaringBitmap> getMatchingFlattenedDocsMap(String jsonPathKey) { | ||
| public Map<String, RoaringBitmap> getMatchingFlattenedDocsMap(String jsonPathKey, @Nullable String filterString) { |
There was a problem hiding this comment.
Not introduced in this PR, but I think we should support json path key with and without $ prefix. See getMatchingFlattenedDocIds() for reference
There was a problem hiding this comment.
Added the support
saurabhd336
force-pushed
the
jsonExtractIndexFilterSupport
branch
from
cb36a1b to
8e45c63
Compare
saurabhd336
force-pushed
the
jsonExtractIndexFilterSupport
branch
from
2581212 to
3771339
Compare
|
@Jackie-Jiang updated the PR to let As for the nested exclusive predicates, I think my understanding was incorrect. We can't infact support nested exclusive predicates (see https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java#L106). Turns out, the contract is for the predicates to maintain context, and therefore trying to solve nested exclusive predicates by flipping results of individual flattened doc id bitmaps would be wrong. Since our usecase does not require nested exclusive predicates, I removed that part entirely. Do have a look ty! |
Jackie-Jiang
added
feature
documentation
release-notes
3 participants
This PR depends on #12532
Allows contextual filtering of documents when using
jsonExtractIndex. EgjsonExtractIndex(jsonField, '$.arrField[*].f1', 'INT_ARRAY', '0', '"$.arrField[*].f2" > 2')returns[3, 0]i.e. Only return matching array elements that match the given json predicate.