Repository navigation
fix(io): keep null lists of structs when projecting Arrow data - #4065
Merged
Merged
Conversation
Contributor
Author
|
@Fokko requesting review |
Contributor
Author
|
Hi, checking whether someone could take a look at this. It fixes #3833: projecting Arrow data turned null |
Contributor
Author
|
@kevinjqliu could you have a look at this one? It fixes #3833, where projecting Arrow data turned null |
|
plus on this, it would be great to get this fix in @kevinjqliu |
The projection visitor rebuilds a list of structs from its offsets alone, which cannot represent a null list, so every null list<struct<...>> was turned into an empty list. On the write path this is persisted into the Parquet file and the null is lost for every reader. Rebuild the list from the original validity and offsets buffers instead, which also keeps the original array offset so sliced batches stay correct. Closes apache#3833
DG47
force-pushed
the
fix/null-list-of-structs
branch
from
November 10, 2026 15:08
f37fc5c to
6df3d03
Compare
Contributor
Author
|
@Fokko request you to review this |
Fokko
self-requested a review
November 11, 2026 21:05
Fokko
approved these changes
Nov 11, 2026
| # assert arrow_table["col_list_with_struct"].to_pylist() == [None, [{'test': 1}]] | ||
| # Once https://github.com/apache/arrow/issues/38809 has been fixed | ||
| assert arrow_table["col_list_with_struct"].to_pylist() == [[], [{"test": 1}]] | ||
| assert arrow_table["col_list_with_struct"].to_pylist() == [None, [{"test": 1}]] |
Member
|
Related tests and fixes |
krishan711
added a commit
to krishan711/iceberg-python
that referenced
this pull request
Nov 12, 2026
…of-structs * upstream/main: (78 commits) fix(io): keep null lists of structs when projecting Arrow data (apache#4065) Skip unknown nested fields when resolving Avro read schemas (apache#4289) fix(expressions): reject non-finite floats on conversion to Decimal (apache#4287) fix: handle negative-scale Decimals in increment, decrement and truncate_decimal (apache#4285) Match NaN partition values with `IsNaN` in partition predicates (apache#4281) Fix append when a data column is named like a partition helper column (apache#4344) Widen decimal literal precision in scan filters (apache#4348) Add row-lineage support with Spark tests (apache#3972) Serialize purgeRequested as a lowercase boolean in REST drop_table (apache#4341) Validate each branch against its own commits on retry (apache#4317) Core: Compare upsert rows column by column instead of cell by cell (apache#4334) Core: Write added files under the partition spec they were planned with (apache#4338) Core: Check whether a failed last commit attempt landed before cleanup (apache#4339) REST: Use the standard User-Agent format (apache#4330) SQL: Accept an existing SQLAlchemy engine in SqlCatalog (apache#4336) Expose catalog labels on Table (apache#4321) Respect REST endpoint capabilities when checking view existence (apache#4337) Reject existing ref names in create_branch and create_tag (apache#4319) Assert every ref a transaction writes, not just the first (apache#4315) feat: send Idempotency-Key on REST mutations when the server advertises it (apache#4310) ... # Conflicts: # pyiceberg/io/pyarrow.py # tests/io/test_pyarrow.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3833
Rationale for this change
ArrowProjectionVisitor.listrebuilds a list whose element is a struct withpa.LargeListArray.from_arrays(list_array.offsets, value_array). The offsets alone cannot express a null list, so every nulllist<struct<...>>came out as an empty list. The visitor runs on both the read and the write path; on write the empty list is persisted into the Parquet file, so the null is gone for every reader, not only for pyiceberg.Reproduction from the issue (pyarrow 25.0.1):
This PR rebuilds the list with
pa.Array.from_buffers, reusing the validity and offsets buffers (and the array offset) of the original list and swapping in the projected element array. Passingmask=list_array.is_null()tofrom_arrayswould fix the unsliced case, but Arrow rejects a mask together with sliced offsets (Null bitmap with offsets slice not supported) forLargeListArray, so sliced batches would start raising. Reusing the buffers keeps the array offset and avoids that.The integration test
test_null_list_and_mapasserted the corrupted value with the correct assertion commented out; the correct assertion is restored.Are these changes tested?
test__to_requested_schema_null_list_of_structsintests/io/test_pyarrow.py, parametrized overpa.list_andpa.large_list, covering a full batch and a sliced one. It fails onmain(At index 2 diff: [] != None) and passes with this change.tests/integration/test_reads.py::test_null_list_and_map.make testandmake lintpass locally.Are there any user-facing changes?
Null lists of structs are now preserved on read and write instead of being turned into empty lists.