Repository navigation
fix: derive custom nullability for spark map_from_entries - #19274
Merged
comphead merged 1 commit intoDec 12, 2025
Merged
Conversation
kszucs
pushed a commit
to kszucs/datafusion
that referenced
this pull request
Jun 1, 2026
## Which issue does this PR close? Closes # (none: prerequisite for apache/datafusion-comet#2706; follow-up to apache#17779 and apache#19274). ## Rationale for this change Spark's default mapKeyDedupPolicy is EXCEPTION, raising SparkRuntimeException with error class DUPLICATED_MAP_KEY on duplicate map keys. The existing Spark map_from_entries / map_from_arrays UDFs silently kept the last occurrence, forcing downstream engines like datafusion-comet to fall back to Spark. ## What changes are included in this PR? Duplicate map keys in Spark map_from_entries, map_from_arrays, and str_to_map now raise [DUPLICATED_MAP_KEY] Duplicate map key {key} was found, matching Spark's default behaviour and error class. ## Are these changes tested? Yes, via sqllogictest assertions covering the new error across the affected Spark map UDFs. ## Are there any user-facing changes? Yes. Duplicate keys now raise [DUPLICATED_MAP_KEY] under the default policy instead of silently collapsing to the last occurrence. No new config keys, no API changes. --------- Co-authored-by: Krishna Sudarshan J <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
map_from_entriesneed to have custom nullability #19161Rationale for this change
What changes are included in this PR?
map_from_entriesnow usesreturn_field_from_argsto handle nullabilityAre these changes tested?
Are there any user-facing changes?