Repository navigation
Conversation
…ULLs When a column of List/LargeList/Map/FixedSizeList is absent from a data file (schema evolution added it later, no initial-default), the missing-column NULL-fill causes: 'unexpected target column type List(...)'. Emit an all-NULL array of exactly the target nested type via new_null_array (preserves nested Field defs + field-id metadata). Matches iceberg-java BaseParquetReaders.defaultReader: absent + optional -> ParquetValueReaders.nulls() (NULL per row, never an empty collection). Closes apache#2618
00e8b0a to
66d84e0
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
@ewoodbury I'm no longer working on iceberg-rust but this is a pretty nefarious error as well if you want to work with @mbutrovich to get it through! It fell through the cracks. |
|
sure, happy to take a look! |
What changes are included in this PR?
When a nested column of List/LargeList/Map/FixedSizeList is absent from a data file (schema evolution added it later, no initial-default), the missing-column NULL-fill causes: 'unexpected target column type List(...)'.
Emit NULL matching the target type (follows what iceberg-java does).
Are these changes tested?
Yes - we run local unit tests with lists and maps to confirm that if omitted from a data file but present in the table schema we'll properly return them with NULL fills.