Skip to content

[Multi-stage] Optimize group key generation #12394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jackie-Jiang merged 1 commit into from
Feb 13, 2024
Merged

[Multi-stage] Optimize group key generation #12394

Jackie-Jiang merged 1 commit into from
Feb 13, 2024

Conversation

@Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented Feb 10, 2024

  • Use primitive type for single key column generation
  • For 2 key columns, combine 2 dictionary ids into a long and use long as the key to generate the ids
  • For multiple key columns, combine dictionary ids into a list and use it as the key to generate the ids
  • Support BigDecimal column as key

The algorithm is the same as NoDictionaryMultiColumnGroupKeyGenerator

@Jackie-Jiang Jackie-Jiang added enhancement bugfix performance multi-stage Related to the multi-stage query engine labels Feb 10, 2024
@codecov-commenter
Copy link

codecov-commenter commented Feb 10, 2024 •
edited
Loading

Codecov Report

Attention: 174 lines in your changes are missing coverage. Please review.

Comparison is base (43dadbf) 61.73% compared to head (fceadd0) 61.71%.

Files Patch % Lines
...upby/NoDictionaryMultiColumnGroupKeyGenerator.java 44.68% 45 Missing and 7 partials ⚠️
.../operator/groupby/OneFloatKeyGroupIdGenerator.java 0.00% 31 Missing ⚠️
...e/operator/groupby/OneLongKeyGroupIdGenerator.java 0.00% 31 Missing ⚠️
...operator/groupby/OneDoubleKeyGroupIdGenerator.java 59.37% 8 Missing and 5 partials ⚠️
...me/operator/groupby/MultiKeysGroupIdGenerator.java 70.00% 9 Missing and 3 partials ⚠️
.../query/aggregation/groupby/utils/ValueToIdMap.java 0.00% 8 Missing ⚠️
...time/operator/groupby/TwoKeysGroupIdGenerator.java 80.95% 3 Missing and 5 partials ⚠️
.../query/aggregation/groupby/utils/FloatToIdMap.java 57.14% 3 Missing ⚠️
...e/query/aggregation/groupby/utils/LongToIdMap.java 57.14% 3 Missing ⚠️
...time/operator/groupby/GroupIdGeneratorFactory.java 70.00% 2 Missing and 1 partial ⚠️
... and 5 more
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #12394      +/-   ##
============================================
- Coverage     61.73%   61.71%   -0.03%     
  Complexity      207      207              
============================================
  Files          2428     2436       +8     
  Lines        132828   133069     +241     
  Branches      20545    20607      +62     
============================================
+ Hits          82007    82118     +111     
- Misses        44811    44921     +110     
- Partials       6010     6030      +20
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration <0.01% <0.00%> (-0.01%) ⬇️
integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration2 0.00% <0.00%> (ø)
java-11 61.67% <57.66%> (+<0.01%) ⬆️
java-21 61.55% <57.66%> (-0.07%) ⬇️
skip-bytebuffers-false 61.69% <57.66%> (-0.02%) ⬇️
skip-bytebuffers-true 61.52% <57.66%> (-0.06%) ⬇️
temurin 61.71% <57.66%> (-0.03%) ⬇️
unittests 61.70% <57.66%> (-0.03%) ⬇️
unittests1 46.87% <57.66%> (-0.05%) ⬇️
unittests2 27.70% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang force-pushed the optimize_group_key_gen branch from 80690c0 to fceadd0 Compare February 10, 2024 10:56
@Jackie-Jiang Jackie-Jiang merged commit dc427fb into apache:master Feb 13, 2024
@Jackie-Jiang Jackie-Jiang deleted the optimize_group_key_gen branch February 13, 2024 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiangfu0 xiangfu0 xiangfu0 approved these changes

Assignees

No one assigned

Labels

bugfix enhancement multi-stage Related to the multi-stage query engine performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Jackie-Jiang @codecov-commenter @xiangfu0