Repository navigation
[native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209
Description
Describe the bug
When switching from native_comet to native_iceberg_compat in Spark 4.0.0, I see some tests fail in VariantShreddingSuite. It appears to be an issue with selecting fields by index from within structs.
[info] == Optimized Logical Plan ==
[info] Project [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info] +- Relation [v#216] parquet
[info]
[info] == Physical Plan ==
[info] *(1) CometColumnarToRow
[info] +- CometProject [variant_get(v, $.a)#211, variant_get(v, $.b)#212L, variant_get(v, $.c)#213, variant_get(v, $.d)#214], [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info] +- CometScan [native_iceberg_compat] parquet [v#216] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/apache-spark-4.0.0/target/tmp/spark-5897c3a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<v:struct<0:int,1:bigint,2:double,3:decimal(9,4)>>
[info]
[info] == Results ==
[info]
[info] == Results ==
[info] !== Correct Answer - 3 == == Spark Answer - 3 ==
[info] !struct<> struct<variant_get(v, $.a):int,variant_get(v, $.b):bigint,variant_get(v, $.c):double,variant_get(v, $.d):decimal(9,4)>
[info] ![1,2,3.3,4.4] [null,null,null,null]
[info] ![5,-6,null,null] [null,null,null,null]
[info] [null,null,null,null] [null,null,null,null]
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
[-][native_iceberg_compat] `VariantShreddingSuite` test failures[/-][+][native_iceberg_compat] `VariantShreddingSuite` test failures with Spark 4.0.0[/+]comphead commentedon Aug 21, 2025
For
native_iceberg_compatthe Comet doesnt use the SparkSchemaAdapter where most of cases already handledparthchandra commentedon Aug 26, 2025
Au contraire
datafusion-comet/native/core/src/parquet/mod.rs
Line 740 in 528822a
datafusion-comet/native/core/src/parquet/parquet_exec.rs
Line 90 in 528822a
parthchandra commentedon Aug 26, 2025
It might be a while till we get to it. arrow-rs has a bunch of issues outstanding for variant. Not sure where we are with the support in parquet.
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0 (#3503)
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0 (apache#3503
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0 (apache#3503
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0 (apache#3503
andygrove commentedon Apr 25, 2026
I am working on this