Repository navigation
fix(core): mark new columns nullable when merging schemas on write - #4833
Merged
ethan-tyler merged 3 commits intoOct 10, 2026
Merged
ethan-tyler merged 3 commits into
ethan-tyler merged 3 commits into
Conversation
Appending with schema_mode="merge" added new columns with their incoming nullability, so a non-nullable column from the batch became non-nullable in the table even though existing rows have no values for it. Later appends of null to that column were then rejected, and for nested structs the table could become unreadable. New fields merged into an existing table are now forced nullable, mirroring Delta Spark which calls schema.asNullable before merging. Adds unit tests for top-level and nested new fields. Closes delta-io#4810 Signed-off-by: Shahzeb Ali <[email protected]>
Shahzeb-DataEngineer
requested review from
ethan-tyler,
ion-elgreco,
roeap and
rtyler
as code owners
October 8, 2026 09:44
ethan-tyler
requested changes
Oct 8, 2026
RecordBatchWriter's merge_and_conform path requires exact type equality in conform_to_schema, so a newly-added nested struct child (merged as nullable) failed the write with SchemaMismatch: the batch's StructArray still declares the child non-nullable. Teach conform_to_schema to cast columns whose types match up to nullability to the merged field type (a data no-op), keeping the hard SchemaMismatch for genuine type changes. Adds regression tests for the nested and top-level new-field cases on the WriteMode::MergeSchema path. Signed-off-by: Shahzeb Ali <[email protected]>
ethan-tyler
enabled auto-merge (rebase)
October 10, 2026 05:46
ethan-tyler
approved these changes
Oct 10, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4833 +/- ##
==========================================
+ Coverage 81.95% 81.96% +0.01%
==========================================
Files 200 200
Lines 74551 74677 +126
Branches 74551 74677 +126
==========================================
+ Hits 61100 61211 +111
- Misses 9872 9884 +12
- Partials 3579 3582 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Appending with
schema_mode="merge"added new columns with the nullability declared by the incoming batch. A non-nullable column in the batch became non-nullable in the table schema even though existing rows have no values for it — so a later append ofnullto that column was rejected by delta-rs's own NOT NULL check, and for fields nested inside an existing struct the table could become unreadable (ArrowTypeError: struct fields don't match).This change forces genuinely new fields to nullable when
merge_arrow_schemapreserves new fields (the schema-merge write paths), mirroring Delta Lake Spark, which callsschema.asNullablebefore merging.merge_delta_struct(used by ADD COLUMN, where nullability is an explicit DDL choice) is intentionally left unchanged.Closes #4810.
Testing
merge_arrow_schema_new_top_level_field_is_nullableandmerge_arrow_schema_new_nested_field_is_nullablecargo test -p deltalake-core --lib kernel::schema::cast: 33 passed, 0 failedcargo fmtcleanAI usage disclosure
Per CONTRIBUTING.md: I used an AI coding assistant while working on this fix. I read the issue, traced the write path (
operations/write/plan.rs→merge_arrow_schema), reviewed allmerge_arrow_*/merge_delta_structcall sites to scope the change so ADD COLUMN semantics are untouched, and verified by compiling the crate and running the unit tests (including the two new ones). Happy to adjust the approach if maintainers prefer the fix applied elsewhere.