Skip to content

Commit e8e3670

Browse files
buraksennthorrester
authored andcommitted
feat(parquet): clip nested wrappers during schema pruning (apache#24315)
## Which issue does this PR close? - Closes apache#24122. ## Rationale for this change Parquet nested schema pruning currently descends through `Struct`, `List`, and `LargeList`. Narrowing casts through `ListView`, `LargeListView`, `FixedSizeList`, or `Dictionary` therefore read physical leaves that the cast never consumes. The issue also listed `RunEndEncoded`, but its casts do not use DataFusion's name-based nested cast path. Arrow may map nested struct fields positionally, so clipping those values could change results; they must remain a full read. ## What changes are included in this PR? Extend nested schema pruning through matching `ListView`, `LargeListView`, `FixedSizeList`, and `Dictionary` wrappers while preserving safe full-read fallbacks. ## Are these changes tested? Yes, with new and existing tests. ## Are there any user-facing changes? No.
1 parent f660ee5 commit e8e3670

1 file changed

Lines changed: 390 additions & 92 deletions

File tree

0 commit comments

Comments
 (0)