Repository navigation
Commit cd3a218
fix: preserve range partitioning through joins (apache#23584)
- Closes apache#23450.
partitioned inner joins without hash repartitioning. However, join
output still downgraded `Partitioning::Range` to `UnknownPartitioning`,
so downstream joins and aggregates could not reuse the preserved range
layout and inserted avoidable hash repartitions.
- Preserve `Partitioning::Range` in `adjust_right_output_partitioning`.
- Shift right-side range-ordering column indexes into the joined schema
while retaining split points and sort options.
- Add unit coverage for compound range keys and non-default sort
options.
- Update SQL logic test plans for nested range joins and a range join
feeding an aggregate.
Yes.
- `./dev/rust_lint.sh`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-physical-plan
test_adjust_right_output_partitioning_preserves_range`
- `cargo test --profile=ci --test sqllogictests --
range_partitioning.slt`
- `ulimit -n 10240 && RUST_BACKTRACE=1 cargo test --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli --workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
Yes. Physical plans can preserve proven range partitioning through
joins, allowing compatible downstream joins and aggregates to avoid
unnecessary hash repartitioning. There are no public API changes.1 parent e98fb60 commit cd3a218
2 files changed
Lines changed: 76 additions & 24 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
| 37 | + | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| |||
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
70 | | - | |
| 71 | + | |
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| |||
145 | 146 | | |
146 | 147 | | |
147 | 148 | | |
148 | | - | |
149 | | - | |
150 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
151 | 162 | | |
152 | 163 | | |
153 | 164 | | |
| |||
2149 | 2160 | | |
2150 | 2161 | | |
2151 | 2162 | | |
2152 | | - | |
| 2163 | + | |
2153 | 2164 | | |
2154 | 2165 | | |
2155 | 2166 | | |
| |||
3244 | 3255 | | |
3245 | 3256 | | |
3246 | 3257 | | |
| 3258 | + | |
| 3259 | + | |
| 3260 | + | |
| 3261 | + | |
| 3262 | + | |
| 3263 | + | |
| 3264 | + | |
| 3265 | + | |
| 3266 | + | |
| 3267 | + | |
| 3268 | + | |
| 3269 | + | |
| 3270 | + | |
| 3271 | + | |
| 3272 | + | |
| 3273 | + | |
| 3274 | + | |
| 3275 | + | |
| 3276 | + | |
| 3277 | + | |
| 3278 | + | |
| 3279 | + | |
| 3280 | + | |
| 3281 | + | |
| 3282 | + | |
| 3283 | + | |
| 3284 | + | |
| 3285 | + | |
| 3286 | + | |
| 3287 | + | |
| 3288 | + | |
| 3289 | + | |
| 3290 | + | |
| 3291 | + | |
| 3292 | + | |
| 3293 | + | |
| 3294 | + | |
| 3295 | + | |
| 3296 | + | |
| 3297 | + | |
| 3298 | + | |
| 3299 | + | |
| 3300 | + | |
| 3301 | + | |
| 3302 | + | |
| 3303 | + | |
| 3304 | + | |
3247 | 3305 | | |
3248 | 3306 | | |
3249 | 3307 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
496 | 496 | | |
497 | 497 | | |
498 | 498 | | |
499 | | - | |
500 | | - | |
501 | | - | |
| 499 | + | |
| 500 | + | |
502 | 501 | | |
503 | 502 | | |
504 | 503 | | |
| |||
509 | 508 | | |
510 | 509 | | |
511 | 510 | | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
518 | 515 | | |
519 | 516 | | |
520 | 517 | | |
| |||
589 | 586 | | |
590 | 587 | | |
591 | 588 | | |
592 | | - | |
593 | | - | |
594 | | - | |
| 589 | + | |
| 590 | + | |
595 | 591 | | |
596 | 592 | | |
597 | 593 | | |
| |||
601 | 597 | | |
602 | 598 | | |
603 | 599 | | |
604 | | - | |
605 | | - | |
606 | | - | |
607 | | - | |
608 | | - | |
609 | | - | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
610 | 604 | | |
611 | 605 | | |
612 | 606 | | |
| |||
0 commit comments