Repository navigation
Better optimization of consecutive keys in aggregation - #57872
Merged
alexey-milovidov merged 6 commits intoDec 27, 2023
Merged
alexey-milovidov merged 6 commits into
alexey-milovidov merged 6 commits into
Conversation
Contributor
|
This is an automated comment for commit 27bcdbe with description of existing statuses. It's updated for the latest CI running ⏳ Click here to open a full report in a separate page Successful checks
|
CurtizJ
force-pushed
the
optimize-aggregation-consecutive-keys
branch
from
December 16, 2023 03:36
a4adabf to
5faf5e9
Compare
Member
Author
CurtizJ
force-pushed
the
optimize-aggregation-consecutive-keys
branch
from
December 18, 2023 13:57
904b764 to
10bc2db
Compare
CurtizJ
marked this pull request as ready for review
December 18, 2023 16:50
antonio2368
reviewed
Dec 19, 2023
| bool check(const Key & key) const { return value.first == key; } | ||
|
|
||
| bool hasOnlyOneValue() const { return found && misses == 1; } | ||
| UInt64 getMisses() const { return misses; } |
Member
There was a problem hiding this comment.
what's the point of getter if fields are public and they don't have getters?
Co-authored-by: Antonio Andelic <[email protected]>
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Dec 28, 2023
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Dec 28, 2023
25 tasks done
5 tasks
pull Bot
pushed a commit
to Lobinson1/ClickHouse
that referenced
this pull request
Jul 30, 2026
TTLAggregationAlgorithm built its Aggregator::Params with min_chunk_bytes_for_parallel_parsing in the argument slot that expects min_hit_rate_to_use_consecutive_keys_optimization. The former is a byte threshold defaulting to 10 MiB, the latter a hit rate in [0, 1] defaulting to 0.5, so the threshold arrived as 10485760.0. Aggregator::executeImpl compares cache_hit_rate (at most 1.0) against that threshold, so use_cache was always false, Method::StateNoCache was selected, and StateImpl<false>::hasOnlyOneValueSinceLastReset() returns false unconditionally. That value gates the equal-key-range batching in executeAggregateInstructions, so addBatchSinglePlace was never taken and the consecutive keys optimization was permanently disabled for TTL GROUP BY aggregation. This is the path that benefits most from it: execute() calls calculateAggregates() once per run of equal keys, so every executeOnBlock receives a single-key chunk. Introduced by ClickHouse#57872, which added the parameter and used the correct setting at every other call site; the narrowing conversion that would have flagged it was later silenced by the static_cast<float> from ClickHouse#94643. Present since v23.12.1.1368-stable. Results are not affected, only performance. There is no parallel parsing in this file, so the now-unused min_chunk_bytes_for_parallel_parsing extern is removed. Measured on the added test (200000 rows, 100 keys, one aggregating TTL merge): ProfileEvents['AggregationOptimizedEqualRangesOfKeys'] goes from 0 to 588 while count() and sum(v) stay identical. Closes: ClickHouse#112075
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Optimized aggregation in some cases.
Detailed description:
There are two optimizations in this PR:
addBatchSinglePlaceinstead ofaddBatchin case when we have only one unique key in block.