Skip to content

Commit 60c24ce

Browse files
schenksjclaude
andcommitted
docs(delta): capture apache#79 investigation notes in CometScanRule gate comment
Removed the gate as an empirical probe to capture the real failure (Utf8 <= Int32 on b.d > 0 constraint check against CM-name table with b: STRUCT<c: STRING, d: INT>), then reinstated it after one hypothesis attempt did not resolve the issue. The expanded comment records what was verified empirically (Delta writes nested struct children with PHYSICAL names in CM-name parquet files), what was tried (also physicalising required_schema's nested children), and what remains for follow-up investigation (DataFusion default SchemaAdapter behavior on nested struct field-name mismatches). No functional change. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 5496bfb commit 60c24ce

1 file changed

Lines changed: 29 additions & 11 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 29 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -656,15 +656,33 @@ case class CometScanRule(session: SparkSession)
656656
val scanWithMappedSchema = withDeltaColumnMappingMetadata(scanExec)
657657

658658
// Targeted fallback: when the table is column-mapped AND the scan's required schema
659-
// contains complex (Array/Map/Struct) columns, decline Comet acceleration. The
660-
// metadata-attach path above covers most column-mapped reads, but the CometScanExec
661-
// [native_delta_compat] fallback route that `CometDeltaNativeScan.convert` leaves in
662-
// place when it bails out doesn't consistently thread nested-type physical names
663-
// through to `CometParquetFileFormat`. For scalar-only column-mapped reads the
664-
// CometDeltaNativeScan serde handles them correctly; for complex-type reads the
665-
// fallback produces "Invalid comparison operation: Utf8 <= Int32" downstream.
666-
// Fixes DeltaDropColumnSuite "drop column with constraints" (struct / array access
667-
// through CHECK constraints on a column-mapped table).
659+
// contains complex (Array/Map/Struct) columns, decline Comet acceleration. Both the
660+
// CometDeltaNativeScan path AND the CometScanExec [native_delta_compat] fallback
661+
// produce "Invalid argument error: Invalid comparison operation: Utf8 <= Int32" on
662+
// DeltaDropColumnSuite "drop column with constraints" -- a CHECK constraint
663+
// referencing `b.d` (INT) on a CM-name table with `b: STRUCT<c: STRING, d: INT>`.
664+
//
665+
// Investigation 2026-05-13 (still open):
666+
// * Empirically verified: Delta DOES write nested-struct children with PHYSICAL
667+
// names in CM-name parquet files (probed via stock Delta 3.3.2 + pyarrow). So
668+
// `data_schema` arriving at the native planner with physical nested names IS
669+
// correct -- it matches the file.
670+
// * `withDeltaColumnMappingMetadata` re-attaches `delta.columnMapping.physicalName`
671+
// metadata to every nested StructField (recurses into Struct/Array/Map).
672+
// * `CometDeltaNativeScan.physicaliseNestedTypesOnly` substitutes physical names
673+
// into data_schema's nested children but keeps required_schema's nested children
674+
// logical.
675+
// * Hypothesis A (physicalise required_schema's nested children too, planner-side
676+
// rename then handles top-level): did NOT fix the failure. Same Utf8 <= Int32.
677+
// * `GetStructField` serializes by ORDINAL not name, so the wrong-field hypothesis
678+
// via name-matching inside the struct doesn't fit cleanly.
679+
// * Next step: instrument the native planner's `data_filters` rewrite path and
680+
// the DataFusion default schema-adapter's behavior on nested struct fields with
681+
// name mismatches -- need to confirm whether the adapter casts mismatched types
682+
// vs. silently passes the wrong column up.
683+
//
684+
// Until rooted, the gate keeps both paths declining and Spark+Delta handles complex
685+
// column-mapped reads correctly via vanilla fallback.
668686
val metadataConfig = DeltaReflection.extractMetadataConfiguration(r).getOrElse(Map.empty)
669687
val isColumnMapped = metadataConfig
670688
.get("delta.columnMapping.mode")
@@ -681,8 +699,8 @@ case class CometScanRule(session: SparkSession)
681699
withInfo(
682700
scanExec,
683701
"Native Delta scan declines column-mapped tables with complex (Array/Map/Struct) " +
684-
"columns in the scan output -- fallback reader path doesn't consistently " +
685-
"remap physical names through to the native expression evaluator.")
702+
"columns in the scan output -- nested-type physical name remapping is not " +
703+
"yet correct (Item #79 deferred follow-up).")
686704
return None
687705
}
688706
}

0 commit comments

Comments
 (0)