perf(sql): support advanced aggregate functions in parallel GROUP BY#4097
Merged
perf(sql): support advanced aggregate functions in parallel GROUP BY#4097
Conversation
1ebca60 to
1d67bf3
Compare
This was referenced Jan 4, 2024
1d67bf3 to
8f187ca
Compare
4649f88 to
2412ab6
Compare
ideoma
reviewed
Jan 10, 2024
.../main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIPv4GroupByFunction.java
Show resolved
Hide resolved
core/src/main/java/io/questdb/griffin/engine/groupby/GroupByAllocator.java
Show resolved
Hide resolved
ideoma
reviewed
Jan 10, 2024
core/src/main/java/io/questdb/griffin/engine/groupby/GroupByAllocator.java
Show resolved
Hide resolved
ideoma
reviewed
Jan 10, 2024
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
…3' into puzpuzpuz_parallel_group_by_pt3
Contributor
Author
Updated Javadoc in ea4e17b |
dd07850 to
35412db
Compare
35412db to
b6660cb
Compare
ideoma
reviewed
Jan 11, 2024
core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByNotKeyedRecordCursor.java
Outdated
Show resolved
Hide resolved
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
...c/main/java/io/questdb/griffin/engine/functions/groupby/CountDistinctIntGroupByFunction.java
Show resolved
Hide resolved
This reverts commit b6660cb.
ideoma
previously approved these changes
Jan 11, 2024
Contributor
Author
|
@ideoma many thanks for the thoughtful review! I've addressed all pending comments. PTAL |
Collaborator
[PR Coverage check]😍 pass : 984 / 1242 (79.23%) file detail
|
6 tasks
ideoma
approved these changes
Jan 11, 2024
This was referenced Oct 2, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The core idea is to use a specialized allocator for aggregate functions with additional state. That's to amortize the cost of frequent alloc/free calls. The allocator is thread-safe, supports best-effort free operation, and frees the memory fully only when it's closed. Considering that the aggregate functions grow their state as the power of 2, the max theoretical memory overhead should be around 2x (that's only with very active data structure to thread migration). See
GroupByAllocatorfor more details.Migrated aggregate functions use new flyweight off-heap data structures backed by the allocator and store the data structure pointers in the GROUP BY map. The data structures can be found in the
GroupByCharSink,GroupByLongHashSetandGroupByIntHashSetclasses.Includes parallel GROUP BY migration for
min(str)/max(str)functions andcount_distinct()function for LONG, INT and IPv4 types.Also includes the following:
ClickBench results on my 4c/8t machine (hot runs):
Microbenchmark results: