Skip to content

Commit 881b871

Browse files
committed
fix: report map builder errors in Spark's row order and trim the per-batch validation
The map_from_arrays validator checked every row's key and value lengths before the kernel looked for a duplicate key, so a duplicate in row 0 followed by a length mismatch in row 1 raised _LEGACY_ERROR_TEMP_2128 where Spark raises DUPLICATED_MAP_KEY. The kernel already checks lengths and duplicates row by row, in Spark's order, so both validators now return early unless a row that gets built holds a NULL key; that row is bound to fail, so only then are the rows up to it walked, lengths first. The kernel's length-mismatch message is translated to SparkError::MapKeyValueDiffSizes like its duplicate-key message, with a pin test. map_from_entries first rules out the common case with one AND and a popcount (a NULL key only under NULL entries), and the walk reuses the duplicate from HashSet::replace instead of cloning every key. The sliced-list workaround now rebases only a list whose first offset is nonzero, zero-copy (shifted offsets and a values slice), links apache/datafusion#25419, and says to delete it once Comet picks up apache/datafusion#25431. An all-scalar call is evaluated once and returned as a scalar, as the kernel alone does, instead of building number_rows rows. Cleanups: derive Default for the wrappers and drop their unused new(), read key nulls with logical_nulls(), drop the unreachable LargeList and FixedSizeList arms and row-count check, drop the clippy allow (clippy is clean without it), parse the policy into the typed ConfigOptions field instead of set_str, and inline session_config_options() at its only caller. The Rust tests that only exercised the upstream kernel are gone; new ones pin the orderings above, a NULL key in a row whose values are NULL or that holds a NULL entry, and the all-scalar result.
1 parent 88c877b commit 881b871

3 files changed

Lines changed: 416 additions & 525 deletions

File tree

‎native/core/src/execution/jni_api.rs‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -759,13 +759,12 @@ fn prepare_datafusion_session_context(
759759
session_config.set_str("datafusion.execution.parquet.reorder_filters", "true");
760760
}
761761

762-
// `map_from_arrays`, `map_from_entries` and `str_to_map` build their maps with the
763-
// duplicate-key policy Spark's `ArrayBasedMapBuilder` uses. DataFusion spells the same
764-
// setting `datafusion.spark.map_key_dedup_policy` and takes the same `EXCEPTION` /
765-
// `LAST_WIN` values. Set before the `spark.comet.datafusion.*` testing escape hatch
766-
// pass-through below, so an explicit override of the DataFusion key still wins.
762+
// The duplicate-key policy of the native map constructors, which DataFusion spells
763+
// `datafusion.spark.map_key_dedup_policy` with the same `EXCEPTION` / `LAST_WIN` values (see
764+
// the map_funcs expression audit). Set before the `spark.comet.datafusion.*` testing escape
765+
// hatch pass-through below, so an explicit override of the DataFusion key still wins.
767766
if let Some(policy) = spark_config.get(SPARK_MAP_KEY_DEDUP_POLICY) {
768-
session_config = session_config.set_str("datafusion.spark.map_key_dedup_policy", policy);
767+
session_config.options_mut().spark.map_key_dedup_policy = policy.parse()?;
769768
}
770769

771770
// Pass through DataFusion configs from Spark.

‎native/core/src/execution/planner.rs‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3666,14 +3666,6 @@ impl PhysicalPlanner {
36663666
}
36673667
}
36683668

3669-
/// The session's `ConfigOptions`, so a kernel that reads one sees what
3670-
/// `prepare_datafusion_session_context` set rather than DataFusion's defaults. The map
3671-
/// builders read `datafusion.spark.map_key_dedup_policy` this way, which Comet forwards from
3672-
/// `spark.sql.mapKeyDedupPolicy`.
3673-
fn session_config_options(&self) -> Arc<ConfigOptions> {
3674-
Arc::clone(self.session_ctx.copied_config().options())
3675-
}
3676-
36773669
fn create_scalar_function_expr(
36783670
&self,
36793671
expr: &ScalarFunc,
@@ -3800,7 +3792,10 @@ impl PhysicalPlanner {
38003792
fun_expr,
38013793
args.to_vec(),
38023794
Arc::new(Field::new(fun_name, data_type.clone(), true)),
3803-
self.session_config_options(),
3795+
// The session's options rather than DataFusion's defaults, so a kernel that reads one
3796+
// (the map constructors read `datafusion.spark.map_key_dedup_policy`) sees what
3797+
// `prepare_datafusion_session_context` set.
3798+
Arc::clone(self.session_ctx.copied_config().options()),
38043799
));
38053800

38063801
// DF53 changed some UDFs (e.g. md5) to return StringViewArray at execution

0 commit comments

Comments
 (0)