Repository navigation
Native Parquet scan reads nested fields by position when field ids no longer match their names #6192
Description
Describe the bug
With spark.sql.parquet.fieldId.read.enabled=true, the native scan reads a nested struct by position whenever the requested struct has the same field names, in the same order and with the same types, as the file's struct but different field ids. Spark resolves those fields by id. Schema evolution that drops a nested column and adds it back under the same name produces exactly this, since the new column gets a new id, and Comet then returns the dropped column's old values where Spark returns null.
Steps to reproduce
On main at 67803a7 with the default Spark 4.1 profile and Comet enabled:
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
spark.conf.set("spark.sql.parquet.fieldId.read.enabled", "true")
def withId(id: Int) = new MetadataBuilder().putLong("parquet.field.id", id).build()
def xy(xId: Int, yId: Int) = new StructType()
.add("x", LongType, true, withId(xId))
.add("y", LongType, true, withId(yId))
spark.createDataFrame(
spark.sparkContext.parallelize(Seq(Row(Row(1L, 2L))), 1),
new StructType().add("s", xy(1, 2), true, withId(10)))
.write.mode("overwrite").parquet("/tmp/relabel")
// x was dropped and added back, so it now carries id 3
spark.read
.schema(new StructType().add("s", xy(3, 2), true, withId(10)))
.parquet("/tmp/relabel")
.show()| File → requested | Spark | Comet |
|---|---|---|
s<x (id 1), y (id 2)> → s<x (id 3), y (id 2)> |
{null, 2} |
{1, 2} |
s<x (id 1), y (id 2)> → s<x (id 2), y (id 1)> |
{2, 1} |
{1, 2} |
The same two shapes inside array<struct> or a map value |
read by id | read by position |
Root x (id 1), y (id 2) → x (id 2), y (id 1) |
{2, 1} |
{2, 1} |
CometNativeScanExec is in the plan for every row, checked against Spark 4.1.3.
Expected behavior
The same answers as Spark. Nested fields that carry a field id are matched by id, so a requested id the file does not have reads as null.
Additional context
is_pure_structural_narrowing declines any target that carries field ids (schema_adapter.rs#L122-L126), so these columns always get a CometCastColumnExpr. Its evaluate then takes the relabel shortcut (cast_column.rs#L278) because types_differ_only_in_field_names (cast_column.rs#L41) compares names, types and nullability but not PARQUET:field_id. The file's array comes back relabelled in place.
The shortcut dates to #3536, and 1.0.0 already sent field ids to the native scan, so 1.0.0 is probably affected too. I haven't run it. The native Iceberg scan is not affected, because it runs the same adapter with use_field_id off (iceberg_scan.rs#L233).
#5654 fixes it. Its once-per-file FieldMapping only allows the relabel shortcut when the mapping is positional, and every shape above matches Spark on that branch.
Activity