Skip to content

Commit bc89e2c

Browse files
committed
fix: bind equality deletes case-sensitively and check nested defaults on the task schema
iceberg-rust now rejects a case-insensitive name that matches two columns when it binds an equality delete's predicate to the task schema, so a table with `id` and `ID` failed any scan whose task schema held both. The names come from that schema, so Comet marks its tasks case-sensitive. The check for V3 default values read only the query's schema. While nested schema pruning is off, the scan reads every nested field of a projected column, and iceberg-rust fails a nested field with a default that a data file lacks. The check now also covers the projected columns of the task schema.
1 parent 793e320 commit bc89e2c

5 files changed

Lines changed: 142 additions & 27 deletions

File tree

‎native/core/src/execution/planner.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2739,7 +2739,11 @@ fn parse_file_scan_tasks_from_common(
27392739
.with_partition_spec(partition_spec)
27402740
.with_name_mapping(name_mapping)
27412741
.with_unified_partition_type(unified_partition_type_for_task)
2742-
.with_case_sensitive(false)
2742+
// The only consumer is the binding of an equality delete's predicate, whose
2743+
// column names iceberg-rust takes from the task schema. They match it exactly,
2744+
// and a case-insensitive binding rejects a schema with two names that differ
2745+
// only by case.
2746+
.with_case_sensitive(true)
27432747
// Plaintext StandardKeyMetadata forwarded verbatim from the JVM; decoded by
27442748
// iceberg-rust with no KMS unwrap. None for unencrypted data files.
27452749
.with_key_metadata(proto_task.key_metadata.clone().map(Vec::into_boxed_slice))

‎spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1283,6 +1283,24 @@ object IcebergReflection extends Logging {
12831283
columns.asScala.flatMap(walkFieldForDefault).toList
12841284
}
12851285

1286+
/**
1287+
* [[columnsWithInitialDefault]] for only the top-level columns of `schema` whose ids are in
1288+
* `columnIds`. A scan reads every nested field of such a column that its schema holds, so this
1289+
* is for the schema the scan reads when that is wider than the query's, as the table's is while
1290+
* nested schema pruning is off.
1291+
*/
1292+
def columnsWithInitialDefault(schema: Any, columnIds: Set[Int]): List[String] = {
1293+
import scala.jdk.CollectionConverters._
1294+
val columns =
1295+
getMethod(schema.getClass, "columns").invoke(schema).asInstanceOf[java.util.List[_]]
1296+
columns.asScala
1297+
.filter(column => columnIds.contains(fieldIdOf(column)))
1298+
.flatMap {
1299+
walkFieldForDefault
1300+
}
1301+
.toList
1302+
}
1303+
12861304
private def walkFieldForDefault(field: Any): List[String] = {
12871305
import scala.jdk.CollectionConverters._
12881306
val name = getMethod(field.getClass, "name").invoke(field).asInstanceOf[String]

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 28 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -743,17 +743,41 @@ case class CometScanRule(session: SparkSession)
743743
false
744744
}
745745

746+
// Projected roots are matched by field ID so historical snapshots still identify renamed
747+
// columns. Both schema checks below use them.
748+
val projectedDataColumns = scanExec.output.filterNot(_.isMetadataCol)
749+
val resolver = session.sessionState.conf.resolver
750+
val projectedFieldIds = projectedDataColumns.map { attr =>
751+
metadata.globalFieldIdMapping.collectFirst {
752+
case (fieldName, fieldId) if resolver(fieldName, attr.name) => fieldId
753+
}
754+
}
755+
val resolvedProjectedFieldIds = projectedFieldIds.flatten.toSet
756+
val hasUnresolvedProjectedFieldIds = projectedFieldIds.exists(_.isEmpty)
757+
746758
// Column default values are a V3 feature, so V1/V2 tables cannot have them and need no
747759
// check. Only inspect V3 tables. iceberg-rust does not synthesize a V3 initial-default for
748-
// a column absent from a data file, so a scan projecting such a column must fall back. A V3
749-
// table implies an Iceberg version new enough to expose initialDefault(), so a reflection
750-
// failure here is unexpected and also falls back rather than risk a crash.
760+
// a column absent from a data file (it fails a nested one), so a scan reading such a
761+
// column must fall back. That is the query's schema and, while nested schema pruning is
762+
// off, the whole of each projected column in the task schema, which holds nested fields
763+
// the query's does not. A V3 table implies an Iceberg version new enough to expose
764+
// initialDefault(), so a reflection failure here is unexpected and also falls back rather
765+
// than risk a crash.
751766
val defaultValuesSupported =
752767
if (!formatVersion.exists(_ >= 3)) {
753768
true
754769
} else {
755770
try {
756-
val defaulted = IcebergReflection.columnsWithInitialDefault(metadata.scanSchema)
771+
val readDefaults = metadata.taskSchema.toSeq.flatMap { schema =>
772+
if (hasUnresolvedProjectedFieldIds) {
773+
IcebergReflection.columnsWithInitialDefault(schema)
774+
} else {
775+
IcebergReflection.columnsWithInitialDefault(schema, resolvedProjectedFieldIds)
776+
}
777+
}
778+
val defaulted =
779+
(IcebergReflection.columnsWithInitialDefault(
780+
metadata.scanSchema) ++ readDefaults).distinct
757781
if (defaulted.nonEmpty) {
758782
fallbackReasons += "Iceberg column(s) with V3 default values are not yet " +
759783
s"supported by Comet's native reader: ${defaulted.mkString(", ")}"
@@ -769,18 +793,6 @@ case class CometScanRule(session: SparkSession)
769793
}
770794
}
771795

772-
// Projected roots are matched by field ID so historical snapshots still identify renamed
773-
// columns. Both schema checks below use them.
774-
val projectedDataColumns = scanExec.output.filterNot(_.isMetadataCol)
775-
val resolver = session.sessionState.conf.resolver
776-
val projectedFieldIds = projectedDataColumns.map { attr =>
777-
metadata.globalFieldIdMapping.collectFirst {
778-
case (fieldName, fieldId) if resolver(fieldName, attr.name) => fieldId
779-
}
780-
}
781-
val resolvedProjectedFieldIds = projectedFieldIds.flatten.toSet
782-
val hasUnresolvedProjectedFieldIds = projectedFieldIds.exists(_.isEmpty)
783-
784796
// iceberg-rust applies name mappings only at the root (#6790). Nested columns can
785797
// disappear during projection, or reach the adapter with physical field names that
786798
// cannot distinguish a mapped alias from a genuinely missing field. Schema history

‎spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1224,6 +1224,80 @@ class CometIcebergNativeSuite
12241224
}
12251225
}
12261226

1227+
// iceberg-rust fails to read a nested field that has a default value and that a data file lacks.
1228+
// While nested schema pruning is off, the scan reads every nested field of a projected column,
1229+
// including one the query prunes away, so the check on defaults must cover the task schema.
1230+
test("nested field with a default value falls back when the scan reads it") {
1231+
assume(icebergAvailable, "Iceberg not available in classpath")
1232+
assume(icebergVersionAtLeast(1, 11), "Nested column defaults require Iceberg 1.11+")
1233+
1234+
withTempIcebergDir { warehouseDir =>
1235+
withSQLConf(
1236+
"spark.sql.catalog.test_cat" -> "org.apache.iceberg.spark.SparkCatalog",
1237+
"spark.sql.catalog.test_cat.type" -> "hadoop",
1238+
"spark.sql.catalog.test_cat.warehouse" -> warehouseDir.getAbsolutePath,
1239+
CometConf.COMET_ENABLED.key -> "true",
1240+
CometConf.COMET_EXEC_ENABLED.key -> "true",
1241+
CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
1242+
val table = "test_cat.db.nested_default"
1243+
spark.sql(
1244+
s"CREATE TABLE $table (id INT, s STRUCT<a: INT>) USING iceberg " +
1245+
"TBLPROPERTIES ('format-version' = '3')")
1246+
spark.sql(
1247+
s"INSERT INTO $table VALUES (1, named_struct('a', 1)), (2, named_struct('a', 2))")
1248+
addIcebergIntColumnWithDefault(
1249+
loadIcebergTable(spark, "test_cat", "db", "nested_default"),
1250+
"b",
1251+
7,
1252+
parent = Some("s"))
1253+
spark.sql(s"REFRESH TABLE $table")
1254+
1255+
// The query prunes s.b, so the pruned task schema omits it.
1256+
checkIcebergNativeScan(s"SELECT id, s.a FROM $table ORDER BY id")
1257+
// With pruning off the scan reads s.b too, and the file lacks it.
1258+
withSQLConf(CometConf.COMET_ICEBERG_NESTED_SCHEMA_PRUNING_ENABLED.key -> "false") {
1259+
checkIcebergNativeScanFallback(
1260+
s"SELECT id, s.a FROM $table ORDER BY id",
1261+
"nested default read by the unpruned task schema")
1262+
}
1263+
checkIcebergNativeScanFallback(
1264+
s"SELECT id, s.b FROM $table ORDER BY id",
1265+
"nested default read by the query")
1266+
spark.sql(s"DROP TABLE $table")
1267+
}
1268+
}
1269+
}
1270+
1271+
// iceberg-rust binds an equality delete's key by name, and rejects a case-insensitive name that
1272+
// matches two columns. The delete's key is taken from the task schema, spelled exactly as in it,
1273+
// so the binding is case sensitive. The query reads neither column, as the batch adaptation
1274+
// after iceberg-rust rejects a read of two columns that differ only by case.
1275+
test("equality delete on a column whose name differs from another only by case") {
1276+
assume(icebergAvailable, "Iceberg not available in classpath")
1277+
1278+
withTempIcebergDir { warehouseDir =>
1279+
withSQLConf(
1280+
"spark.sql.catalog.test_cat" -> "org.apache.iceberg.spark.SparkCatalog",
1281+
"spark.sql.catalog.test_cat.type" -> "hadoop",
1282+
"spark.sql.catalog.test_cat.warehouse" -> warehouseDir.getAbsolutePath,
1283+
SQLConf.CASE_SENSITIVE.key -> "true",
1284+
CometConf.COMET_ENABLED.key -> "true",
1285+
CometConf.COMET_EXEC_ENABLED.key -> "true",
1286+
CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
1287+
val table = "test_cat.db.eq_delete_case"
1288+
spark.sql(
1289+
s"CREATE TABLE $table (id INT, ID STRING, v STRING) USING iceberg " +
1290+
"TBLPROPERTIES ('format-version' = '2')")
1291+
spark.sql(s"INSERT INTO $table VALUES (1, 'a', 'x'), (2, 'b', 'y'), (3, 'c', 'z')")
1292+
commitEqualityDelete("test_cat", "db", "eq_delete_case", "id", 2, warehouseDir)
1293+
1294+
// The pruned task schema holds only `v` and the key `id`. The full one holds both columns.
1295+
checkPrunedAndFullReads(s"SELECT v FROM $table ORDER BY v")
1296+
spark.sql(s"DROP TABLE $table")
1297+
}
1298+
}
1299+
}
1300+
12271301
// The key `s.k` is nested, which iceberg-rust cannot look up, so the scan must fall back.
12281302
test("equality delete keyed on a nested field falls back to Spark") {
12291303
assume(icebergAvailable, "Iceberg not available in classpath")

‎spark/src/test/scala/org/apache/comet/CometIcebergTestBase.scala‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -124,13 +124,16 @@ trait CometIcebergTestBase { this: CometTestBase =>
124124

125125
/**
126126
* Adds an `int` column whose initial and write defaults are both `defaultValue`, through
127-
* `UpdateSchema.addColumn(name, type, Literal)`. Requires Iceberg 1.10+ and a v3 table. Callers
128-
* must `REFRESH TABLE` afterwards.
127+
* `UpdateSchema.addColumn(name, type, Literal)`, or to the struct `parent` through
128+
* `addColumn(parent, name, type, Literal)`. Requires Iceberg 1.10+ and a v3 table. Callers must
129+
* `REFRESH TABLE` afterwards.
129130
*/
130131
protected def addIcebergIntColumnWithDefault(
131132
icebergTable: AnyRef,
132133
columnName: String,
133-
defaultValue: Int): Unit = {
134+
defaultValue: Int,
135+
parent: Option[String] = None): Unit = {
136+
val typeClass = IcebergReflection.loadClass("org.apache.iceberg.types.Type")
134137
val intType = IcebergReflection
135138
.loadClass("org.apache.iceberg.types.Types$IntegerType")
136139
.getMethod("get")
@@ -139,18 +142,22 @@ trait CometIcebergTestBase { this: CometTestBase =>
139142
.loadClass("org.apache.iceberg.expressions.Expressions")
140143
.getMethod("lit", classOf[Object])
141144
.invoke(null, Integer.valueOf(defaultValue))
145+
val literalClass = IcebergReflection.loadClass("org.apache.iceberg.expressions.Literal")
142146
val update = IcebergReflection
143147
.loadClass("org.apache.iceberg.Table")
144148
.getMethod("updateSchema")
145149
.invoke(icebergTable)
146150
val updateSchemaClass = IcebergReflection.loadClass("org.apache.iceberg.UpdateSchema")
147-
updateSchemaClass
148-
.getMethod(
149-
"addColumn",
150-
classOf[String],
151-
IcebergReflection.loadClass("org.apache.iceberg.types.Type"),
152-
IcebergReflection.loadClass("org.apache.iceberg.expressions.Literal"))
153-
.invoke(update, columnName, intType, literal)
151+
parent match {
152+
case Some(struct) =>
153+
updateSchemaClass
154+
.getMethod("addColumn", classOf[String], classOf[String], typeClass, literalClass)
155+
.invoke(update, struct, columnName, intType, literal)
156+
case None =>
157+
updateSchemaClass
158+
.getMethod("addColumn", classOf[String], typeClass, literalClass)
159+
.invoke(update, columnName, intType, literal)
160+
}
154161
updateSchemaClass.getMethod("commit").invoke(update)
155162
}
156163

0 commit comments

Comments
 (0)