Repository navigation
SUM(x + C) rewrite - #15017
Merged
Merged
SUM(x + C) rewrite#15017
Conversation
duckdb-draftbot
marked this pull request as draft
November 28, 2024 15:37
Mytherin
marked this pull request as ready for review
November 28, 2024 15:37
duckdb-draftbot
marked this pull request as draft
November 28, 2024 19:23
Mytherin
marked this pull request as ready for review
November 28, 2024 19:23
Contributor
|
Randomly selected... 🤣 |
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
Dec 27, 2024
SUM(x + C) rewrite (duckdb/duckdb#15017)
github-actions Bot
pushed a commit
to duckdb/duckdb-r
that referenced
this pull request
Dec 27, 2024
SUM(x + C) rewrite (duckdb/duckdb#15017)
github-actions Bot
added a commit
to duckdb/duckdb-r
that referenced
this pull request
Dec 27, 2024
SUM(x + C) rewrite (duckdb/duckdb#15017) Co-authored-by: krlmlr <[email protected]>
This was referenced Mar 3, 2026
github-merge-queue Bot
pushed a commit
to apache/datafusion
that referenced
this pull request
Mar 14, 2026
…0749) ## Which issue does this PR close? - Part of #18489 - Closes #20180 - Closes #15524 - Replaces #20665 ## Rationale for this change I [want DataFusion to be the fastest parquet engine on ClickBench](#18489). One of the queries where DataFusion is significantly slower is Query 29 which has a very strange pattern of many aggregate functions that are offset by a constant: https://github.com/apache/datafusion/blob/0ca9d6586a43c323525b2e299448e0f1af4d6195/benchmarks/queries/clickbench/queries/q29.sql#L4 This is not a pattern I have ever seen in a real query, but it seems like the engine currently at the top of the ClickBench leaderboard has a special case for this pattern. ClickHouse probably does too. See - duckdb/duckdb#15017 - Discussion on #15524 Thus I reluctantly conclude that we should have one too. ## What changes are included in this PR? This is an alternate to my first attempt. - #20665 In particular, since this is such a ClickBench specific rule, I wanted to 1. Minimize the downstream API / upgrade impact (aka not change existing APIs) 2. Optimize performance for the case where this rewrite will not apply (most times) 1. Add a rewrite `SUM(expr + scalar)` --> `SUM(expr) + scalar*COUNT(expr)` 3. Tests for same Note there are quite a few other ideas to potentially make this more general on #15524 but I am going with the simple thing of making it work for the usecase we have in hand (ClickBench) ## Are these changes tested? Yes, new tests are added ## Are there any user-facing changes? Faster performance 🚀 ``` │ QQuery 29 │ 1012.63 ms │ 139.02 ms │ +7.28x faster │ ```
de-bgunter
pushed a commit
to de-bgunter/datafusion
that referenced
this pull request
Mar 24, 2026
…ache#20749) ## Which issue does this PR close? - Part of apache#18489 - Closes apache#20180 - Closes apache#15524 - Replaces apache#20665 ## Rationale for this change I [want DataFusion to be the fastest parquet engine on ClickBench](apache#18489). One of the queries where DataFusion is significantly slower is Query 29 which has a very strange pattern of many aggregate functions that are offset by a constant: https://github.com/apache/datafusion/blob/0ca9d6586a43c323525b2e299448e0f1af4d6195/benchmarks/queries/clickbench/queries/q29.sql#L4 This is not a pattern I have ever seen in a real query, but it seems like the engine currently at the top of the ClickBench leaderboard has a special case for this pattern. ClickHouse probably does too. See - duckdb/duckdb#15017 - Discussion on apache#15524 Thus I reluctantly conclude that we should have one too. ## What changes are included in this PR? This is an alternate to my first attempt. - apache#20665 In particular, since this is such a ClickBench specific rule, I wanted to 1. Minimize the downstream API / upgrade impact (aka not change existing APIs) 2. Optimize performance for the case where this rewrite will not apply (most times) 1. Add a rewrite `SUM(expr + scalar)` --> `SUM(expr) + scalar*COUNT(expr)` 3. Tests for same Note there are quite a few other ideas to potentially make this more general on apache#15524 but I am going with the simple thing of making it work for the usecase we have in hand (ClickBench) ## Are these changes tested? Yes, new tests are added ## Are there any user-facing changes? Faster performance 🚀 ``` │ QQuery 29 │ 1012.63 ms │ 139.02 ms │ +7.28x faster │ ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Suppose there is a query that contains the aggregate
SUM(x + 1), this aggregate can be decomposed intoSUM(x) + COUNT(x). In particular if there are multiple of such clauses, e.g.SUM(x + 1), SUM(x + 2), ...this decomposition is useful as we can avoid computing many aggregates, and instead only need to compute two (SUM(x) and COUNT(x)).Benchmark
As a randomly selected query, let's run this query from ClickBench:
Below is the performance before and after this PR: