Skip to content

Conversation

@Jackie-Jiang
Copy link
Contributor

  • Allow broker to automatically rewrite expensive function to its approximate counterpart
    • DISTINCT_COUNT -> DISTINCT_COUNT_SMART_HLL
    • PERCENTILE -> PERCENTILE_SMART_TDIGEST
  • Add table level override for the rewrite through the query config

Release-Notes

Add broker config pinot.broker.use.approximate.function to turn the feature on (off by default)
Add query config useApproximateFunction to override the broker level config

@Jackie-Jiang Jackie-Jiang added enhancement release-notes Referenced by PRs that need attention when compiling the next release notes labels May 6, 2022
@Jackie-Jiang Jackie-Jiang requested a review from xiangfu0 May 6, 2022 22:59
@Jackie-Jiang Jackie-Jiang force-pushed the automatic_approximate_query_rewrite branch from 82ea0d7 to c036b99 Compare May 7, 2022 00:03
@codecov-commenter
Copy link

codecov-commenter commented May 7, 2022

Codecov Report

Merging #8655 (c036b99) into master (ec42a3a) will increase coverage by 0.05%.
The diff coverage is 78.18%.

@@             Coverage Diff              @@
##             master    #8655      +/-   ##
============================================
+ Coverage     70.35%   70.40%   +0.05%     
  Complexity     4406     4406              
============================================
  Files          1690     1690              
  Lines         88187    88249      +62     
  Branches      13190    13204      +14     
============================================
+ Hits          62042    62135      +93     
+ Misses        21856    21829      -27     
+ Partials       4289     4285       -4     
Flag Coverage Δ
integration1 27.47% <24.54%> (+0.02%) ⬆️
integration2 25.57% <24.54%> (-0.10%) ⬇️
unittests1 66.16% <100.00%> (-0.02%) ⬇️
unittests2 14.35% <55.45%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...va/org/apache/pinot/spi/utils/CommonConstants.java 21.31% <ø> (ø)
...roker/requesthandler/BaseBrokerRequestHandler.java 70.37% <77.77%> (+1.02%) ⬆️
...org/apache/pinot/spi/config/table/QueryConfig.java 90.90% <100.00%> (+37.57%) ⬆️
...nt/local/startree/v2/store/StarTreeDataSource.java 40.00% <0.00%> (-13.34%) ⬇️
...mmon/request/context/predicate/NotEqPredicate.java 61.53% <0.00%> (-7.70%) ⬇️
...transform/function/IsNotNullTransformFunction.java 67.85% <0.00%> (-7.15%) ⬇️
...ot/segment/local/startree/OffHeapStarTreeNode.java 65.90% <0.00%> (-6.82%) ⬇️
...core/startree/operator/StarTreeFilterOperator.java 88.02% <0.00%> (-4.23%) ⬇️
...ava/org/apache/pinot/core/plan/FilterPlanNode.java 80.45% <0.00%> (-1.51%) ⬇️
...perator/filter/SortedIndexBasedFilterOperator.java 85.71% <0.00%> (-0.96%) ⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ec42a3a...c036b99. Read the comment docs.

@Jackie-Jiang Jackie-Jiang merged commit c0056dc into apache:master May 9, 2022
@Jackie-Jiang Jackie-Jiang deleted the automatic_approximate_query_rewrite branch May 9, 2022 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants