Repository navigation
Fix append when a data column is named like a partition helper column - #4344
Merged
Fokko merged 2 commits intoNov 11, 2026
Merged
Conversation
_determine_partitions adds the transformed partition values as columns named _partition_<field name>. A data column with the same name made the column lookup ambiguous and the append raised KeyError. Pick helper names that are not already in the table. Co-authored-by: Isaac <[email protected]>
Co-authored-by: Isaac <[email protected]>
Contributor
Author
Fokko
approved these changes
Nov 11, 2026
Fokko
left a comment
Fokko
left a comment
Contributor
There was a problem hiding this comment.
Sounds like an edge case, but reasonable to add. Thanks @rahulsmahadev
krishan711
added a commit
to krishan711/iceberg-python
that referenced
this pull request
Nov 12, 2026
…of-structs * upstream/main: (78 commits) fix(io): keep null lists of structs when projecting Arrow data (apache#4065) Skip unknown nested fields when resolving Avro read schemas (apache#4289) fix(expressions): reject non-finite floats on conversion to Decimal (apache#4287) fix: handle negative-scale Decimals in increment, decrement and truncate_decimal (apache#4285) Match NaN partition values with `IsNaN` in partition predicates (apache#4281) Fix append when a data column is named like a partition helper column (apache#4344) Widen decimal literal precision in scan filters (apache#4348) Add row-lineage support with Spark tests (apache#3972) Serialize purgeRequested as a lowercase boolean in REST drop_table (apache#4341) Validate each branch against its own commits on retry (apache#4317) Core: Compare upsert rows column by column instead of cell by cell (apache#4334) Core: Write added files under the partition spec they were planned with (apache#4338) Core: Check whether a failed last commit attempt landed before cleanup (apache#4339) REST: Use the standard User-Agent format (apache#4330) SQL: Accept an existing SQLAlchemy engine in SqlCatalog (apache#4336) Expose catalog labels on Table (apache#4321) Respect REST endpoint capabilities when checking view existence (apache#4337) Reject existing ref names in create_branch and create_tag (apache#4319) Assert every ref a transaction writes, not just the first (apache#4315) feat: send Idempotency-Key on REST mutations when the server advertises it (apache#4310) ... # Conflicts: # pyiceberg/io/pyarrow.py # tests/io/test_pyarrow.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
_determine_partitionsadds the transformed partition values to the Arrow table as temporary columns named_partition_<partition field name>. If the data already has a column with that name, the lookup by name is ambiguous andappend()fails:This change picks helper column names that are not already in the table, by prefixing underscores until the name is free. The helper columns are still dropped before the data files are written.
Are these changes tested?
Yes.
test_determine_partitions_with_column_named_like_partition_helperintests/io/test_pyarrow.pypartitions a table that has a_partition_regiondata column and checks that every row lands in the right partition with that column unchanged. It fails without the fix.Are there any user-facing changes?
Bug fix only: appends to such tables now succeed. No API changes.
This pull request and its description were written by Isaac.