Repository navigation
Some existing tests fail if we enable dictionary encoding in FuzzDataGenerator #2636
Description
Describe the bug
CometFuzzTestSuite
- regexp_replace (native_comet, native shuffle) *** FAILED *** (198 milliseconds)
2025-10-23T00:06:26.8480812Z org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 6267.0 failed 1 times, most recent failure: Lost task 4.0 in stage 6267.0 (TID 16053) (localhost executor driver): org.apache.comet.CometNativeException: Unsupported data type Dictionary(Int32, Utf8) for function regexp_replace.
- regexp_replace (native_comet, jvm shuffle) *** FAILED *** (146 milliseconds)
2025-10-23T00:06:27.4005547Z org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 6278.0 failed 1 times, most recent failure: Lost task 3.0 in stage 6278.0 (TID 16095) (localhost executor driver): org.apache.comet.CometNativeException: Unsupported data type Dictionary(Int32, Utf8) for function regexp_replace.
CometMapExpressionSuite
- map_from_arrays *** FAILED *** (416 milliseconds)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3.0 (TID 12) (localhost executor driver): org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) but found Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Dictionary(Int32, Utf8), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) at column index 0
CometArrayExpressionSuite
- array_except - test all types (native Parquet reader) *** FAILED *** (1 second, 834 milliseconds)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 823.0 failed 1 times, most recent failure: Lost task 0.0 in stage 823.0 (TID 2615) (localhost executor driver): org.apache.comet.CometNativeException: called `Result::unwrap()` on an `Err` value: InvalidArgumentError("ListArray expected data type Dictionary(Int32, Utf8) got Utf8 for \"item\"")
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
mbutrovich commentedon Oct 23, 2025
This raises a larger issue of "do we want to support dictionary-encoded string columns in Comet?"
My understanding of the situation:
native_cometscan tries to dictionary-encoded string columns. But I think they then immediately get unpacked by a CopyExec.native_datafusionandnative_iceberg_compatdo not dictionary-encode string columns, but have the option of using Utf8View for similar gains. We do not fully support Utf8View yet. I have a branch with basic support here https://github.com/mbutrovich/datafusion-comet/tree/german_style_strings, and the tracking issue is here Utf8View and BinaryView (i.e., StringView in Arrow, colloquially German-style strings) support #1403My long term goal is to get us to Uft8View support and drop dictionary-encoded query processing since that brings Comet into alignment with DataFusion and Arrow-rs, but I am happy to discuss.
andygrove commentedon Oct 23, 2025
This was also my understanding, but it is not (always) true.
We sometimes inject a
CopyExec, but only for certain operators. We were not injecting one for projections, causing a runtime error for any expressions that do not support dictionary-encoded strings, such asregexp_replace, and there are others.Sounds good to me.