Skip to content

Streamlining #8134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
richardstartin wants to merge 12 commits into from
Closed

Streamlining #8134

richardstartin wants to merge 12 commits into from

Conversation

@richardstartin
Copy link
Member

@richardstartin richardstartin commented Feb 4, 2022 •
edited
Loading

This addresses many inefficiencies which show up in a benchmark of the query engine

  • delaying conversion of int[] to double[] speeds up reading blocks from storage, reduces the size of arrays stored in the DataBlockCache
  • Don't detect function types by throwing exceptions
  • Size TransformFunction results properly.
  • Don't eagerly format error messages in function init

master

Benchmark                                                                            (_intBaseValue)  (_numRows)  Mode  Cnt        Score         Error   Units
BenchmarkFilteredAggregations.testFilteredAggregations                                             0     1500000  avgt    5    14603.463 ±    2802.292   us/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.alloc.rate                              0     1500000  avgt    5       20.648 ±       3.821  MB/sec
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.alloc.rate.norm                         0     1500000  avgt    5   473834.895 ±    4343.609    B/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.churn.G1_Eden_Space                     0     1500000  avgt    5       35.002 ±     301.381  MB/sec
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.churn.G1_Eden_Space.norm                0     1500000  avgt    5   785693.566 ± 6765058.556    B/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.churn.G1_Old_Gen                        0     1500000  avgt    5        7.788 ±      67.060  MB/sec
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.churn.G1_Old_Gen.norm                   0     1500000  avgt    5   174822.896 ± 1505277.856    B/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.count                                   0     1500000  avgt    5        1.000                counts
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.time                                    0     1500000  avgt    5        4.000                    ms
BenchmarkFilteredAggregations.testNonFilteredAggregations                                          0     1500000  avgt    5    31641.121 ±    2374.442   us/op
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.alloc.rate                           0     1500000  avgt    5       23.624 ±       1.729  MB/sec
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.alloc.rate.norm                      0     1500000  avgt    5  1171658.220 ±   20300.958    B/op
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.count                                0     1500000  avgt    5          ≈ 0                counts

branch

Benchmark                                                                      (_intBaseValue)  (_numRows)  Mode  Cnt       Score         Error   Units
BenchmarkFilteredAggregations.testFilteredAggregations                                       0     1500000  avgt    5   14581.370 ±    1927.876   us/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.alloc.rate                        0     1500000  avgt    5      13.738 ±      13.321  MB/sec
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.alloc.rate.norm                   0     1500000  avgt    5  313800.088 ±  289751.441    B/op
BenchmarkFilteredAggregations.testFilteredAggregations:·gc.count                             0     1500000  avgt    5         ≈ 0                counts
BenchmarkFilteredAggregations.testNonFilteredAggregations                                    0     1500000  avgt    5   28648.469 ±    3463.714   us/op
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.alloc.rate                     0     1500000  avgt    5      16.622 ±      26.714  MB/sec
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.alloc.rate.norm                0     1500000  avgt    5  748228.779 ± 1208200.361    B/op
BenchmarkFilteredAggregations.testNonFilteredAggregations:·gc.count                          0     1500000  avgt    5         ≈ 0                counts

master

Benchmark                                                  (_intBaseValue)  (_numRows)  Mode  Cnt      Score      Error  Units
BenchmarkFilteredAggregations.testFilteredAggregations                   0     1500000  avgt    5  14759.558 ±  783.314  us/op
BenchmarkFilteredAggregations.testNonFilteredAggregations                0     1500000  avgt    5  32208.800 ± 1346.069  us/op

branch

Benchmark                                                  (_intBaseValue)  (_numRows)  Mode  Cnt      Score      Error  Units
BenchmarkFilteredAggregations.testFilteredAggregations                   0     1500000  avgt    5  14066.896 ± 2505.217  us/op
BenchmarkFilteredAggregations.testNonFilteredAggregations                0     1500000  avgt    5  28785.087 ± 4468.296  us/op

@richardstartin richardstartin force-pushed the streamlining branch 4 times, most recently from 32ce43b to 33b50ff Compare February 5, 2022 00:04
@codecov-commenter
Copy link

codecov-commenter commented Feb 5, 2022 •
edited
Loading

Codecov Report

Merging #8134 (a06bbb2) into master (8bbf93a) will decrease coverage by 40.76%.
The diff coverage is 56.87%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master    #8134       +/-   ##
=============================================
- Coverage     71.39%   30.62%   -40.77%     
=============================================
  Files          1624     1613       -11     
  Lines         84198    83915      -283     
  Branches      12602    12593        -9     
=============================================
- Hits          60116    25703    -34413     
- Misses        19970    55922    +35952     
+ Partials       4112     2290     -1822
Flag Coverage Δ
integration1 28.88% <56.87%> (+0.02%) ⬆️
integration2 27.63% <55.92%> (-0.04%) ⬇️
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...orm/function/LogicalOperatorTransformFunction.java 69.56% <0.00%> (-26.27%) ⬇️
...che/pinot/segment/spi/AggregationFunctionType.java 0.00% <0.00%> (-97.23%) ⬇️
...ator/transform/function/CaseTransformFunction.java 40.34% <37.34%> (-24.33%) ⬇️
...e/pinot/common/function/TransformFunctionType.java 84.14% <44.44%> (-15.86%) ⬇️
...ot/common/function/FunctionDefinitionRegistry.java 50.00% <50.00%> (-50.00%) ⬇️
...r/transform/function/LiteralTransformFunction.java 44.68% <68.42%> (-32.54%) ⬇️
...a/org/apache/pinot/core/common/DataBlockCache.java 82.14% <87.80%> (-9.97%) ⬇️
...y/aggregation/function/MaxAggregationFunction.java 81.63% <89.47%> (-18.37%) ⬇️
...y/aggregation/function/SumAggregationFunction.java 95.45% <89.47%> (-4.55%) ⬇️
...org/apache/pinot/sql/parsers/CalciteSqlParser.java 66.42% <100.00%> (-21.53%) ⬇️
... and 1128 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8bbf93a...a06bbb2. Read the comment docs.

@richardstartin richardstartin force-pushed the streamlining branch 3 times, most recently from 60964b3 to 47eb343 Compare February 5, 2022 15:59
This was referenced Feb 5, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@richardstartin @codecov-commenter