You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9504942
Browse filesBrowse the repository at this point in the historyBrowse files
feat: support col reference for percentiles (apache#25337)
## Which issue does this PR close?
- Closesapache#24337.
## Rationale for this change
When using a projection as the percentile argument to
`approx_percentile_cont` or `percentile_cont`, planning fails because
only literals are supported.
It makes sense to accept projections that are constant across all
batches but not labelled as literals (although they practically are).
For example, this query cannot be planned, even though the percentile is
constant:
```
SELECT
approx_percentile_cont(y, m) AS median
FROM (
SELECT t.x + 1 as y, 0.5 as m
FROM (
VALUES (10)
) AS t(x)
)
```
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
Introduce `PercentileParam` and `PercentileParamState` to handle column
references or projections for the percentile argument.
Resolution of `percentile` cannot always be done: for example, on an
empty record batch, we won't be able to get the value from the columns.
`PercentileParamState` makes sure the percentile resolution is both
flexible enough to wait to be able to resolve the parameter AND applies
the strict constraints that are required (constant `Float32` or
`Float64`).
The percentile is also included in state fields so that the merge_batch
in the final Accumulator can process without any issues.
## What is the testing strategy for this PR?
This new feature is covered by the `sqllogictest` cases added in
`aggregate.slt`.
Unit tests have also been added to `approx_percentile_cont.rs` and
`percentile_cont.rs`.
## Are there any user-facing changes?
No breaking changes.
## Benchark: ClickBench
| Query | main | annarose_approx_percentile | Change |
| ---------- | ---------- | ---------- | ---------- |
| QQuery 0 | 0.73 ms | 0.79 ms | 1.08x slower |
| QQuery 1 | 8.10 ms | 8.91 ms | 1.10x slower |
| QQuery 2 | 27.90 ms | 30.59 ms | 1.10x slower |
| QQuery 3 | 23.27 ms | 23.89 ms | no change |
| QQuery 4 | 157.25 ms | 157.70 ms | no change |
| QQuery 5 | 197.65 ms | 201.26 ms | no change |
| QQuery 6 | 0.72 ms | 0.67 ms | +1.08x faster |
| QQuery 7 | 8.20 ms | 8.28 ms | no change |
| QQuery 8 | 230.84 ms | 215.32 ms | +1.07x faster |
| QQuery 9 | 296.27 ms | 312.37 ms | 1.05x slower |
| QQuery 10 | 49.12 ms | 47.66 ms | no change |
| QQuery 11 | 58.86 ms | 55.26 ms | +1.07x faster |
| QQuery 12 | 198.45 ms | 192.95 ms | no change |
| QQuery 13 | 285.28 ms | 280.02 ms | no change |
| QQuery 14 | 195.67 ms | 204.80 ms | no change |
| QQuery 15 | 184.29 ms | 180.74 ms | no change |
| QQuery 16 | 434.20 ms | 457.81 ms | 1.05x slower |
| QQuery 17 | 476.52 ms | 427.84 ms | +1.11x faster |
| QQuery 18 | 912.71 ms | 881.69 ms | no change |
| QQuery 19 | 18.14 ms | 19.41 ms | 1.07x slower |
| QQuery 20 | 456.87 ms | 463.23 ms | no change |
| QQuery 21 | 468.67 ms | 482.15 ms | no change |
| QQuery 22 | 892.28 ms | 896.13 ms | no change |
| QQuery 23 | 2520.93 ms | 2558.49 ms | no change |
| QQuery 24 | 34.14 ms | 33.50 ms | no change |
| QQuery 25 | 99.14 ms | 99.83 ms | no change |
| QQuery 26 | 35.25 ms | 34.51 ms | no change |
| QQuery 27 | 467.74 ms | 484.36 ms | no change |
| QQuery 28 | 1931.71 ms | 1945.50 ms | no change |
| QQuery 29 | 29.16 ms | 30.35 ms | no change |
| QQuery 30 | 214.69 ms | 209.17 ms | no change |
| QQuery 31 | 196.43 ms | 199.65 ms | no change |
| QQuery 32 | 583.78 ms | 614.36 ms | 1.05x slower |
| QQuery 33 | 1078.73 ms | 1145.79 ms | 1.06x slower |
| QQuery 34 | 1073.15 ms | 1117.56 ms | no change |
| QQuery 35 | 182.87 ms | 171.52 ms | +1.07x faster |
| QQuery 36 | 46.27 ms | 47.18 ms | no change |
| QQuery 37 | 24.27 ms | 24.03 ms | no change |
| QQuery 38 | 29.58 ms | 29.76 ms | no change |
| QQuery 39 | 85.29 ms | 86.14 ms | no change |
| QQuery 40 | 8.23 ms | 8.41 ms | no change |
| QQuery 41 | 8.12 ms | 8.47 ms | no change |
| QQuery 42 | 6.72 ms | 6.81 ms | no change |
| Benchmark Summary | |
| ---- | ---- |
| Total Time (main) | 14238.19ms |
| Total Time (annarose_approx_percentile) | 14404.87ms |
| Average Time (main) | 331.12ms |
| Average Time (annarose_approx_percentile) | 335.00ms |
| Queries Faster | 5 |
| Queries Slower | 8 |
| Queries with No Change | 30 |
| Queries with Failure | 0 |
## Memory profiling for clickbench_partitioned
### this branch
|Query | Time (ms) | Peak RSS | Peak Commit | Major Page Faults|
|--------|----------------|-------------|--------------|-------------|
|0|5.43|84.0 MB|171.8 MB|0|
|1|11.45|119.6 MB|313.9 MB|0|
|2|27.44|240.8 MB|1536.1 MB|0|
|3|30.52|412.3 MB|1227.9 MB|0|
|4|173.88|2.8 GB|3.2 GB|0|
|5|193.60|2.5 GB|2.9 GB|0|
|6|3.36|93.0 MB|167.9 MB|0|
|7|10.66|128.1 MB|318.8 MB|0|
|8|219.80|3.4 GB|4.1 GB|0|
|9|268.65|2.4 GB|3.2 GB|0|
|10|45.84|780.3 MB|1809.8 MB|0|
|11|52.68|788.7 MB|1800.9 MB|0|
|12|183.80|2.5 GB|2.9 GB|0|
|13|266.55|3.0 GB|3.5 GB|0|
|14|191.93|2.4 GB|3.2 GB|0|
|15|185.03|3.0 GB|3.3 GB|0|
|16|422.42|6.4 GB|7.1 GB|0|
|17|404.75|6.1 GB|6.9 GB|0|
|18|861.29|12.2 GB|13.1 GB|0|
|19|19.20|362.3 MB|1118.0 MB|24|
|20|442.22|1171.1 MB|1429.2 MB|0|
|21|407.55|1836.8 MB|2.1 GB|0|
|22|763.73|2.5 GB|2.8 GB|0|
|23|2313.37|4.8 GB|5.5 GB|0|
|24|33.92|569.7 MB|959.2 MB|5|
|25|82.99|798.0 MB|1517.9 MB|0|
|26|35.10|599.1 MB|986.8 MB|0|
|27|392.34|1234.9 MB|1455.8 MB|0|
|28|1714.18|2.5 GB|2.7 GB|0|
|29|28.43|229.9 MB|1497.1 MB|0|
|30|191.47|2.2 GB|3.3 GB|0|
|31|182.56|2.8 GB|3.7 GB|0|
|32|1263.74|13.3 GB|16.2 GB|0|
|33|958.16|10.7 GB|10.9 GB|5|
|34|956.65|10.6 GB|10.9 GB|0|
|35|171.08|2.1 GB|2.6 GB|0|
|36|47.33|452.0 MB|800.8 MB|0|
|37|26.89|229.4 MB|544.9 MB|0|
|38|33.62|270.9 MB|503.8 MB|0|
|39|90.83|756.9 MB|1062.6 MB|0|
|40|11.41|208.3 MB|433.3 MB|0|
|41|11.47|170.3 MB|393.8 MB|0|
|42|10.48|142.8 MB|265.1 MB|0|
### main
|Query | Time (ms) | Peak RSS | Peak Commit |Major Page Faults|
|----------|--------------|------------|----------------|------------|
|0|4.76|94.7 MB|171.5 MB|0|
|1|11.22|120.7 MB|296.7 MB|0|
|2|32.05|244.2 MB|1533.8 MB|0|
|3|24.33|417.7 MB|1229.2 MB|0|
|4|175.85|2.8 GB|3.2 GB|0|
|5|207.07|2.5 GB|3.0 GB|0|
|6|3.46|94.3 MB|171.6 MB|0|
|7|11.08|129.4 MB|323.5 MB|0|
|8|223.08|3.4 GB|4.1 GB|0|
|9|288.63|2.4 GB|3.4 GB|0|
|10|48.79|792.1 MB|1767.0 MB|0|
|11|53.32|760.8 MB|1779.6 MB|0|
|12|189.17|2.6 GB|3.1 GB|0|
|13|274.96|3.0 GB|3.7 GB|0|
|14|202.74|2.6 GB|3.3 GB|0|
|15|208.83|3.0 GB|3.4 GB|0|
|16|576.37|6.3 GB|7.1 GB|0|
|17|623.87|6.5 GB|7.3 GB|0|
|18|1140.54|11.3 GB|13.1 GB|0|
|19|21.20|370.8 MB|1124.9 MB|26|
|20|507.91|1149.3 MB|1364.8 MB|0|
|21|479.85|1824.8 MB|2.1 GB|0|
|22|914.26|2.3 GB|2.7 GB|0|
|23|2757.47|4.6 GB|5.3 GB|0|
|24|41.85|591.0 MB|999.1 MB|0|
|25|91.00|763.3 MB|1500.8 MB|0|
|26|38.90|560.1 MB|980.6 MB|0|
|27|506.23|1319.5 MB|1550.4 MB|0|
|28|2119.47|2.6 GB|2.7 GB|0|
|29|34.79|213.7 MB|1267.2 MB|0|
|30|218.63|2.1 GB|3.1 GB|0|
|31|212.42|2.8 GB|3.7 GB|0|
|32|821.63|13.0 GB|16.1 GB|0|
|33|1162.42|10.5 GB|10.7 GB|14|
|34|1212.91|10.4 GB|10.6 GB|0|
|35|184.71|2.0 GB|2.5 GB|0|
|36|58.35|469.5 MB|845.9 MB|0|
|37|30.78|240.2 MB|545.1 MB|0|
|38|35.43|287.3 MB|485.4 MB|0|
|39|92.89|730.8 MB|1074.5 MB|0|
|40|11.87|204.7 MB|420.5 MB|0|
|41|11.43|171.2 MB|401.5 MB|0|
|42|10.57|146.6 MB|275.6 MB|0|
0 commit comments