Skip to content

Commit f84c6ba

Browse files
schenksjclaude
andcommitted
fix(delta): correct nested-CM read for column-mapped tables (apache#79)
Closes apache#79. Removes the gate that declined column-mapped tables with complex (Array/Map/Struct) columns in scan output. Root cause: under Catalyst nested-column pruning, the native parquet reader was sent a `data_schema` whose `b` struct kept all its file children (e.g. `b: struct<col-c STRING, col-d INT>`), while `required_schema` had only the projected child (`b: struct<d INT>`). DataFusion's ParquetSource projects by top-level index only, so the output struct came back with 2 children. Spark's catalyst-baked `GetStructField(b, ordinal=0)` (computed against the *pruned* schema where d is at position 0) then picked the file-shape's position 0 - `col-c` (STRING) - producing the "Invalid comparison Utf8 <= Int32" mismatch on `b.d > 0` constraint evaluation. Fix (CometDeltaNativeScan.convert): for each field in data_schema that is also in required_schema, replace its data type with required's already-physicalised pruned shape. ParquetSource then reads only the required nested children and emits a struct matching what Spark's plan expects. Non-required top-level columns keep their full physicalised shape (harmless since projection_vector doesn't reference them). Required_schema gets a new helper `physicaliseRequiredField` / `physicaliseDataTypePreserving` that walks `req`'s tree pairing nodes with the snapshot by logical name to apply physical names, while preserving req's pruning shape. The data-schema helper (`physicaliseNestedTypesOnly`) was wrong for required_schema because it replaced the whole data type with the snapshot's full shape, losing pruning. Verified: - DeltaDropColumnSuite "drop column with constraints" and "drop column with constraints - map element" both PASS (was FAILING before the gate was added). - DeltaDropColumnSuite + DeletionVectorsSuite + DeltaColumnMappingSuite show 16 failures, ALL of which also fail on stock Delta 3.3.2 when running the same 3 suites together (verified upstream: 17 failures, identical set minus the 2B-row test we skip). Net: zero Comet-introduced regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent a83ecb3 commit f84c6ba

2 files changed

Lines changed: 98 additions & 57 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 17 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -662,55 +662,24 @@ case class CometScanRule(session: SparkSession)
662662
// (the merge is a no-op when the snapshot schema has no physical-name entries).
663663
val scanWithMappedSchema = withDeltaColumnMappingMetadata(scanExec)
664664

665-
// Targeted fallback: when the table is column-mapped AND the scan's required schema
666-
// contains complex (Array/Map/Struct) columns, decline Comet acceleration. Both the
667-
// CometDeltaNativeScan path AND the CometScanExec [native_delta_compat] fallback
668-
// produce "Invalid argument error: Invalid comparison operation: Utf8 <= Int32" on
669-
// DeltaDropColumnSuite "drop column with constraints" -- a CHECK constraint
670-
// referencing `b.d` (INT) on a CM-name table with `b: STRUCT<c: STRING, d: INT>`.
665+
// #79 fixed 2026-05-13: nested-type physical name remapping was incorrect under
666+
// Catalyst's nested column pruning. `physicaliseNestedTypesOnly` (still used for
667+
// data_schema, which is the FULL relation schema) replaced the field's whole
668+
// data type with the snapshot's full data type. When applied to required_schema
669+
// -- which Spark may have pruned to a single child like `b: struct<d INT>` --
670+
// the replacement re-introduced the sibling children (e.g. `c STRING`), so the
671+
// native ParquetSource emitted a 2-child struct where Spark's catalyst-baked
672+
// GetStructField(b, ordinal=0) expected a 1-child struct. The DataFusion schema
673+
// adapter's by-name nested matching couldn't reconcile the logical/physical name
674+
// mismatch and fell back positionally, picking col-c (STRING) instead of
675+
// col-d (INT). Result: "Utf8 <= Int32" mismatch when the constraint
676+
// `b.d > 0` evaluated.
671677
//
672-
// Investigation 2026-05-13 (still open):
673-
// * Empirically verified: Delta DOES write nested-struct children with PHYSICAL
674-
// names in CM-name parquet files (probed via stock Delta 3.3.2 + pyarrow). So
675-
// `data_schema` arriving at the native planner with physical nested names IS
676-
// correct -- it matches the file.
677-
// * `withDeltaColumnMappingMetadata` re-attaches `delta.columnMapping.physicalName`
678-
// metadata to every nested StructField (recurses into Struct/Array/Map).
679-
// * `CometDeltaNativeScan.physicaliseNestedTypesOnly` substitutes physical names
680-
// into data_schema's nested children but keeps required_schema's nested children
681-
// logical.
682-
// * Hypothesis A (physicalise required_schema's nested children too, planner-side
683-
// rename then handles top-level): did NOT fix the failure. Same Utf8 <= Int32.
684-
// * `GetStructField` serializes by ORDINAL not name, so the wrong-field hypothesis
685-
// via name-matching inside the struct doesn't fit cleanly.
686-
// * Next step: instrument the native planner's `data_filters` rewrite path and
687-
// the DataFusion default schema-adapter's behavior on nested struct fields with
688-
// name mismatches -- need to confirm whether the adapter casts mismatched types
689-
// vs. silently passes the wrong column up.
690-
//
691-
// Until rooted, the gate keeps both paths declining and Spark+Delta handles complex
692-
// column-mapped reads correctly via vanilla fallback.
693-
val metadataConfig = DeltaReflection.extractMetadataConfiguration(r).getOrElse(Map.empty)
694-
val isColumnMapped = metadataConfig
695-
.get("delta.columnMapping.mode")
696-
.exists(m => m != null && !m.equalsIgnoreCase("none"))
697-
if (isColumnMapped) {
698-
def isComplex(dt: DataType): Boolean = dt match {
699-
case _: ArrayType | _: MapType | _: StructType => true
700-
case _ => false
701-
}
702-
val requiredComplex =
703-
scanWithMappedSchema.requiredSchema.fields.exists(f => isComplex(f.dataType))
704-
val outputComplex = scanWithMappedSchema.output.exists(a => isComplex(a.dataType))
705-
if (requiredComplex || outputComplex) {
706-
withInfo(
707-
scanExec,
708-
"Native Delta scan declines column-mapped tables with complex (Array/Map/Struct) " +
709-
"columns in the scan output -- nested-type physical name remapping is not " +
710-
"yet correct (Item #79 deferred follow-up).")
711-
return None
712-
}
713-
}
678+
// Fix: `CometDeltaNativeScan` now uses `physicaliseRequiredField` /
679+
// `physicaliseDataTypePreserving` for required_schema -- walks `req`'s tree,
680+
// pairs each node with the snapshot's metadata to rewrite to physical names,
681+
// and preserves req's pruned shape. data_schema continues to use the
682+
// full-replacement helper (correct because data_schema isn't pruned).
714683

715684
// `__delta_internal_is_row_deleted` and `__delta_internal_row_index` are synthetic
716685
// columns produced ONLY by Delta's own `DeltaParquetFileFormat` reader; Comet's

‎spark/src/main/scala/org/apache/comet/serde/operator/CometDeltaNativeScan.scala‎

Lines changed: 81 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -458,17 +458,89 @@ object CometDeltaNativeScan extends CometOperatorSerde[CometScanExec] with Loggi
458458
StructField(f.name, physicaliseDataType(metaField.dataType), f.nullable, f.metadata)
459459
case None => f
460460
}
461-
// Only `data_schema` (the on-disk parquet schema) takes physical names; keep
462-
// `required_schema` in logical names because the planner's `ColumnMappingFilterRewriter`
463-
// + schema-adapter chain uses the name delta between the two to produce physical reads and
464-
// logical-named outputs. Physicalising required_schema would break downstream operators
465-
// that reference the output by logical name (aggregations, joins, etc.).
466-
val physicalFileDataSchemaFields =
467-
if (columnMappingActive) fileDataSchemaFields.map(physicaliseNestedTypesOnly)
468-
else fileDataSchemaFields
461+
// For `required_schema` we MUST preserve the field's pruned shape (Spark's
462+
// nested column pruning can leave a struct with only the accessed children) while
463+
// still rewriting nested names to their physical equivalents. Using the data-schema
464+
// helper above (which replaces the whole struct with the snapshot's full shape)
465+
// would lose pruning and produce nested children Spark's plan does not expect,
466+
// causing GetStructField ordinals to point at the wrong child. Walks `req`'s tree
467+
// and pairs each node with the corresponding snapshot node by logical name to find
468+
// the physical name; fields not present in the snapshot pass through untouched.
469+
def physicaliseDataTypePreserving(req: DataType, snap: DataType): DataType =
470+
(req, snap) match {
471+
case (rs: StructType, ms: StructType) =>
472+
val snapByLogical = ms.fields.map(f => f.name -> f).toMap
473+
StructType(rs.fields.map { rf =>
474+
snapByLogical.get(rf.name) match {
475+
case Some(mf) =>
476+
val physName =
477+
if (mf.metadata.contains(DeltaReflection.PhysicalNameMetadataKey)) {
478+
mf.metadata.getString(DeltaReflection.PhysicalNameMetadataKey)
479+
} else rf.name
480+
StructField(
481+
physName,
482+
physicaliseDataTypePreserving(rf.dataType, mf.dataType),
483+
rf.nullable,
484+
rf.metadata)
485+
case None => rf
486+
}
487+
})
488+
case (ra: ArrayType, ma: ArrayType) =>
489+
ArrayType(
490+
physicaliseDataTypePreserving(ra.elementType, ma.elementType),
491+
ra.containsNull)
492+
case (rm: MapType, mm: MapType) =>
493+
MapType(
494+
physicaliseDataTypePreserving(rm.keyType, mm.keyType),
495+
physicaliseDataTypePreserving(rm.valueType, mm.valueType),
496+
rm.valueContainsNull)
497+
case _ => req
498+
}
499+
def physicaliseRequiredField(f: StructField): StructField =
500+
physicalByLogicalName.get(f.name) match {
501+
case Some(metaField) =>
502+
StructField(
503+
f.name,
504+
physicaliseDataTypePreserving(f.dataType, metaField.dataType),
505+
f.nullable,
506+
f.metadata)
507+
case None => f
508+
}
509+
// `data_schema` describes what we want the native parquet reader to read from
510+
// the file. Under column mapping, parquet column matching is by PHYSICAL name
511+
// (at every level of nesting). The reader projects by leaf column path -- it
512+
// can read just `b.col-d` even if the file's `b` also has `col-c`. To make
513+
// that happen, we send data_schema with the SAME shape as the required output:
514+
// top-level fields that are required carry the pruned + physicalised nested
515+
// shape; non-required top-level fields keep their full physicalised shape (no
516+
// read attempt is made for them anyway because they don't appear in
517+
// projection_vector). Without this overlay, the reader would emit a struct
518+
// with ALL nested children (full file shape), and upstream GetStructField
519+
// ordinals -- computed by Catalyst against the PRUNED required_schema --
520+
// would pick the wrong child. Manifested as "Invalid comparison Utf8 <= Int32"
521+
// on `b.d > 0` (d is INT, ordinal 0 in pruned `b: struct<d>`, but ordinal 0
522+
// in the file struct is `c` STRING). #79 fix 2026-05-13.
523+
val requiredSchemaFields =
524+
if (columnMappingActive) scan.requiredSchema.fields.map(physicaliseRequiredField)
525+
else scan.requiredSchema.fields
526+
val physicalFileDataSchemaFields = if (columnMappingActive) {
527+
val requiredByName = requiredSchemaFields
528+
.map(f => f.name.toLowerCase(Locale.ROOT) -> f)
529+
.toMap
530+
fileDataSchemaFields.map { f =>
531+
requiredByName.get(f.name.toLowerCase(Locale.ROOT)) match {
532+
// Required asks for this field -- adopt its pruned, physicalised shape so
533+
// the parquet reader projects only the required nested children.
534+
case Some(req) => StructField(f.name, req.dataType, f.nullable, f.metadata)
535+
// Field not required -- physicalise the full snapshot shape (used only if
536+
// some other consumer references it; harmless when projection_vector skips it).
537+
case None => physicaliseNestedTypesOnly(f)
538+
}
539+
}
540+
} else fileDataSchemaFields
469541

470542
val dataSchema = schema2Proto(physicalFileDataSchemaFields)
471-
val requiredSchema = schema2Proto(scan.requiredSchema.fields)
543+
val requiredSchema = schema2Proto(requiredSchemaFields)
472544
val partitionSchema = schema2Proto(relation.partitionSchema.fields)
473545
commonBuilder.addAllDataSchema(dataSchema.toIterable.asJava)
474546
commonBuilder.addAllRequiredSchema(requiredSchema.toIterable.asJava)

0 commit comments

Comments
 (0)