Repository navigation
Commit 341e173
feat: union nested Parquet leaves across projection accesses (apache#24130)
## Which issue does this PR close?
- Closes apache#24121.
## Rationale for this change
When one projected nested column is consumed through multiple narrowing
casts, or through both a narrowing cast and `get_field`, the Parquet
read plan currently falls back to reading the entire root column. The
leaves required by those consumers can be combined, so reading the full
root performs unnecessary I/O.
## What changes are included in this PR?
- Accumulate and deduplicate the leaf offsets required by every
narrowing cast on a root column.
- Add leaves required by `get_field` accesses to the same union.
- Derive the Arrow type emitted by the reader from the merged leaf set
while retaining a full-read fallback for unsupported partial wrapper
shapes.
- Add coverage for disjoint, overlapping, and repeated casts, cast plus
`get_field`, and casts that diverge below `List<Struct>`.
## Are these changes tested?
Yes. The following checks pass:
- `cargo fmt --all -- --check`
- `cargo test -p datafusion-datasource-parquet --lib`
- `cargo test --profile=ci --test sqllogictests --
parquet_nested_schema_pruning.slt`
- `cargo clippy -p datafusion-datasource-parquet --all-targets
--all-features -- -D warnings`
The SQL logic test also verifies that the merged projections return the
expected values and scan fewer bytes than a full-root read.
## Are there any user-facing changes?
No API or SQL behavior changes. Queries with multiple nested accesses to
the same Parquet root can read fewer leaf columns.
(cherry picked from commit f660ee5)1 parent 7d3835c commit 341e173
3 files changed
Lines changed: 287 additions & 79 deletions
File tree
- datafusion
- datasource-parquet/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
131 | 160 | | |
132 | 161 | | |
133 | 162 | | |
| |||
249 | 278 | | |
250 | 279 | | |
251 | 280 | | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
252 | 329 | | |
253 | 330 | | |
254 | 331 | | |
| |||
455 | 532 | | |
456 | 533 | | |
457 | 534 | | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
458 | 567 | | |
459 | 568 | | |
460 | 569 | | |
| |||
0 commit comments