Skip to content

SUM(x + C) rewrite - #15017

Merged
Mytherin merged 8 commits into
duckdb:mainfrom
Mytherin:veryusefuloptimization
Nov 29, 2024
Merged

Mytherin merged 8 commits into
duckdb:mainfrom
Mytherin:veryusefuloptimization

Conversation

@Mytherin

Copy link
Copy Markdown
Member

Suppose there is a query that contains the aggregate SUM(x + 1), this aggregate can be decomposed into SUM(x) + COUNT(x). In particular if there are multiple of such clauses, e.g. SUM(x + 1), SUM(x + 2), ... this decomposition is useful as we can avoid computing many aggregates, and instead only need to compute two (SUM(x) and COUNT(x)).

Benchmark

As a randomly selected query, let's run this query from ClickBench:

SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits;

Below is the performance before and after this PR:

v1.1.3 New
0.48s 0.025s

@duckdb-draftbot
duckdb-draftbot marked this pull request as draft November 28, 2024 15:37
@Mytherin
Mytherin marked this pull request as ready for review November 28, 2024 15:37
@duckdb-draftbot
duckdb-draftbot marked this pull request as draft November 28, 2024 19:23
@Mytherin
Mytherin marked this pull request as ready for review November 28, 2024 19:23
@Mytherin
Mytherin merged commit 3be8f0b into duckdb:main Nov 29, 2024
Mytherin added a commit that referenced this pull request Nov 30, 2024
…low errors (#15040)

Follow-up fix from #15017

Fixes an internally found issue
@Alex-Monahan

Copy link
Copy Markdown
Contributor

Randomly selected... 🤣

@Mytherin
Mytherin deleted the veryusefuloptimization branch December 8, 2024 06:51
krlmlr added a commit to duckdb/duckdb-r that referenced this pull request Dec 27, 2024
github-actions Bot pushed a commit to duckdb/duckdb-r that referenced this pull request Dec 27, 2024
github-actions Bot added a commit to duckdb/duckdb-r that referenced this pull request Dec 27, 2024
github-merge-queue Bot pushed a commit to apache/datafusion that referenced this pull request Mar 14, 2026
…0749)

## Which issue does this PR close?

- Part of #18489 
- Closes #20180
- Closes #15524
- Replaces #20665

## Rationale for this change

I [want DataFusion to be the fastest parquet engine on
ClickBench](#18489). One of
the queries where DataFusion is significantly slower is Query 29 which
has a very strange pattern of many aggregate functions that are offset
by a constant:


https://github.com/apache/datafusion/blob/0ca9d6586a43c323525b2e299448e0f1af4d6195/benchmarks/queries/clickbench/queries/q29.sql#L4

This is not a pattern I have ever seen in a real query, but it seems
like the engine currently at the top of the ClickBench leaderboard has a
special case for this pattern. ClickHouse probably does too. See
- duckdb/duckdb#15017
- Discussion on #15524

Thus I reluctantly conclude that we should have one too. 

## What changes are included in this PR?

This is an alternate to my first attempt. 
 - #20665

In particular, since this is such a ClickBench specific rule, I wanted
to
1. Minimize the downstream API / upgrade impact (aka not change existing
APIs)
2. Optimize performance for the case where this rewrite will not apply
(most times)

1. Add a rewrite `SUM(expr + scalar)` --> `SUM(expr) +
scalar*COUNT(expr)`
3. Tests for same

Note there are quite a few other ideas to potentially make this more
general on #15524 but I am
going with the simple thing of making it work for the usecase we have in
hand (ClickBench)

## Are these changes tested?

Yes, new tests are added

## Are there any user-facing changes?

Faster performance

🚀 
```
│ QQuery 29 │  1012.63 ms │                 139.02 ms │ +7.28x faster │
```
de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
…ache#20749)

## Which issue does this PR close?

- Part of apache#18489 
- Closes apache#20180
- Closes apache#15524
- Replaces apache#20665

## Rationale for this change

I [want DataFusion to be the fastest parquet engine on
ClickBench](apache#18489). One of
the queries where DataFusion is significantly slower is Query 29 which
has a very strange pattern of many aggregate functions that are offset
by a constant:


https://github.com/apache/datafusion/blob/0ca9d6586a43c323525b2e299448e0f1af4d6195/benchmarks/queries/clickbench/queries/q29.sql#L4

This is not a pattern I have ever seen in a real query, but it seems
like the engine currently at the top of the ClickBench leaderboard has a
special case for this pattern. ClickHouse probably does too. See
- duckdb/duckdb#15017
- Discussion on apache#15524

Thus I reluctantly conclude that we should have one too. 

## What changes are included in this PR?

This is an alternate to my first attempt. 
 - apache#20665

In particular, since this is such a ClickBench specific rule, I wanted
to
1. Minimize the downstream API / upgrade impact (aka not change existing
APIs)
2. Optimize performance for the case where this rewrite will not apply
(most times)

1. Add a rewrite `SUM(expr + scalar)` --> `SUM(expr) +
scalar*COUNT(expr)`
3. Tests for same

Note there are quite a few other ideas to potentially make this more
general on apache#15524 but I am
going with the simple thing of making it work for the usecase we have in
hand (ClickBench)

## Are these changes tested?

Yes, new tests are added

## Are there any user-facing changes?

Faster performance

🚀 
```
│ QQuery 29 │  1012.63 ms │                 139.02 ms │ +7.28x faster │
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants