You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9e680c4
Browse filesBrowse the repository at this point in the historyBrowse files
feat: fall back collect_list/collect_set on Spark 4.2 RESPECT NULLS
Spark 4.2 adds an ignoreNulls field to CollectList and CollectSet, and
collect_list(x) RESPECT NULLS sets it to false, keeping null elements. The
native path delegates to SparkCollectList/SparkCollectSet, which always drop
nulls, so it would silently return a different result from Spark. Add a
per-version CometCollectShim that reads ignoreNulls (always true on Spark 3.4
through 4.1, where the field is absent) and fall back to Spark in
getSupportLevel when it is false.
Also rename QueryPlanSerde.hasIncompatibleBufferAgg to hasNativeArrayBufferAgg
to describe what it detects: an aggregate whose native ArrayType state cannot
round-trip Spark's declared BinaryType buffer.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/agg_funcs.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,6 +46,7 @@
46
46
- Spark 4.0.1 (audited 2026-06-24): only structural change is adding `with UnaryLike[Expression]` to the case class (no behavior change).
47
47
- Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.
48
48
- Comet implementation: native side delegates to `datafusion_spark::function::aggregate::collect::SparkCollectList`, which wraps `ArrayAggAccumulator` with `ignore_nulls = true` and converts a final NULL accumulator state to an empty array (matching Spark's `defaultResult`). The native return type is `List(Field, containsNull = true)`, while Spark uses `containsNull = false`. Because nulls are filtered before insertion, no nulls actually appear in the array, so this is a schema-shape difference only and tests using `checkSparkAnswerAndOperator` accept it (same pattern already in use for `collect_set`).
49
+
- Spark 4.2 (preview): `CollectList` and `CollectSet` gain an `ignoreNulls` field (default `true`); `RESPECT NULLS` sets it to `false` and keeps null elements. The native path always drops nulls, so `CometCollectShim` reads the field per Spark version (always `true` on 3.4-4.1) and `CometCollectList` / `CometCollectSet` report `Unsupported` when it is `false`, falling back to Spark.
0 commit comments