Repository navigation
Commit f660ee5
feat: union nested Parquet leaves across projection accesses (apache#24130)
## Which issue does this PR close?
- Closes apache#24121.
## Rationale for this change
When one projected nested column is consumed through multiple narrowing
casts, or through both a narrowing cast and `get_field`, the Parquet
read plan currently falls back to reading the entire root column. The
leaves required by those consumers can be combined, so reading the full
root performs unnecessary I/O.
## What changes are included in this PR?
- Accumulate and deduplicate the leaf offsets required by every
narrowing cast on a root column.
- Add leaves required by `get_field` accesses to the same union.
- Derive the Arrow type emitted by the reader from the merged leaf set
while retaining a full-read fallback for unsupported partial wrapper
shapes.
- Add coverage for disjoint, overlapping, and repeated casts, cast plus
`get_field`, and casts that diverge below `List<Struct>`.
## Are these changes tested?
Yes. The following checks pass:
- `cargo fmt --all -- --check`
- `cargo test -p datafusion-datasource-parquet --lib`
- `cargo test --profile=ci --test sqllogictests --
parquet_nested_schema_pruning.slt`
- `cargo clippy -p datafusion-datasource-parquet --all-targets
--all-features -- -D warnings`
The SQL logic test also verifies that the merged projections return the
expected values and scan fewer bytes than a full-root read.
## Are there any user-facing changes?
No API or SQL behavior changes. Queries with multiple nested accesses to
the same Parquet root can read fewer leaf columns.1 parent 7d3835c commit f660ee5
3 files changed
Lines changed: 287 additions & 79 deletions
File tree
- datafusion
- datasource-parquet/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
131 | 160 | | |
132 | 161 | | |
133 | 162 | | |
| |||
249 | 278 | | |
250 | 279 | | |
251 | 280 | | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
252 | 329 | | |
253 | 330 | | |
254 | 331 | | |
| |||
455 | 532 | | |
456 | 533 | | |
457 | 534 | | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
458 | 567 | | |
459 | 568 | | |
460 | 569 | | |
| |||
0 commit comments