Skip to content

Commit 793e320

Browse files
committed
feat: bump iceberg-rust past #3255 and narrow the nested schema evolution fallback
Moves the iceberg-rust pin to b154603a, which matches a data file's nested fields by field id (apache/iceberg-rust#3255). The fallback added by #6543 now covers only a struct whose current fields were all added after a data file was written, the one shape where the native read still differs from Spark. That check runs on the task schema the scan reads. The fallback for nested columns of name-mapped tables (#6790) stays. With the bump, a pruned read of a map value whose projected leaves all differ from the file's physical names fails in iceberg-rust with "Map field must have exactly 2 fields". Closes #6934
1 parent 6fe73d9 commit 793e320

6 files changed

Lines changed: 189 additions & 128 deletions

File tree

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -217,10 +217,10 @@ The following scenarios will fall back to the JVM Iceberg reader:
217217
- Scans with residual filters using `truncate`, `bucket`, `year`, `month`, `day`, or `hour`
218218
transform functions (partition pruning still works, but row-level filtering of these
219219
transforms falls back)
220-
- Scans that read a struct, array, or map column with a nested field that schema evolution added
221-
or renamed. The native reader cannot yet match such a field to data files written before the
222-
change. The check uses the table's schema history, so the fallback stays after those files are
223-
rewritten
220+
- Scans that read a struct column, or a struct inside an array or map, whose fields were all added
221+
by schema evolution after some data file was written. For such a file the native reader returns a
222+
struct of NULLs or NULL, depending on what else the scan reads, where Spark does not. The check
223+
uses the table's schema history, so the fallback stays after those files are rewritten
224224
- Scans that read a struct, array, or map column from a table with
225225
`schema.name-mapping.default`, including tables imported with `snapshot`, `migrate`, or
226226
`add_files` ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The native reader
@@ -801,7 +801,8 @@ Which entries apply depends on which native paths are enabled:
801801
### Reads that differ from Spark
802802

803803
- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such
804-
a file as NULL, and the native scan returns the values stored in the file
804+
a file as NULL, and the native scan returns the values stored in the file. A read of only some
805+
of a struct's fields fails instead
805806
([#6794](https://github.com/apache/datafusion-comet/issues/6794)).
806807
- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as
807808
NULL, and two columns that swapped names return each other's values

‎native/Cargo.lock‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎native/Cargo.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,8 @@ aws-sdk-sts = { version = "1.114.0", default-features = false }
8585
# Pinned to a revision rather than a release because no published iceberg-rust
8686
# version is on our arrow: 0.10.1 requires arrow/parquet ^58. Move to a plain
8787
# version requirement once a release ships on arrow 59. Bump policy: #5645.
88-
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b" }
89-
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
88+
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "b154603a710d730d2f43f8af4db257cc137237ef" }
89+
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "b154603a710d730d2f43f8af4db257cc137237ef", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
9090
reqsign-core = "3"
9191

9292
[profile.release]

‎spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala‎

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1304,15 +1304,15 @@ object IcebergReflection extends Logging {
13041304
}
13051305

13061306
/**
1307-
* Returns the nested fields of the `fieldIds` columns of `schema` that some schema in the
1308-
* table's history (`table.schemas()`) lacks or names differently. Each is a dotted path with
1309-
* the change, for example `items.element.z (renamed from a)`. A data file written under that
1310-
* older schema lacks the field, or has it under the old name. Fields are matched by id at every
1311-
* level (struct fields, list elements, map keys and values), so a reorder or a type promotion
1312-
* is not a change, and an older schema that lacks the column itself is skipped. Throws on
1307+
* Returns the structs under the `fieldIds` columns of `schema`, at any depth and including the
1308+
* columns themselves, that share none of their fields with the same struct in some schema in
1309+
* the table's history (`table.schemas()`). Each is a dotted path, for example
1310+
* `items.element.s`. A data file written under that older schema has none of the struct's
1311+
* current fields. Structs are matched by field id at every level (struct fields, list elements,
1312+
* map keys and values), so an older schema that lacks the column itself is skipped. Throws on
13131313
* reflection failure so the caller can fall back.
13141314
*/
1315-
def nestedFieldsAddedOrRenamed(table: Any, schema: Any, fieldIds: Set[Int]): Seq[String] = {
1315+
def structsWithNoFieldsInCommon(table: Any, schema: Any, fieldIds: Set[Int]): Seq[String] = {
13161316
import scala.jdk.CollectionConverters._
13171317
val findField = getMethod(schema.getClass, "findField", classOf[Int])
13181318
def fieldById(s: Any, id: Int): Option[Any] =
@@ -1324,31 +1324,33 @@ object IcebergReflection extends Logging {
13241324
.asScala
13251325
.toSeq
13261326
fieldIds.toSeq.flatMap { id =>
1327-
fieldById(schema, id).filter(childFields(_).nonEmpty).toSeq.flatMap { field =>
1327+
fieldById(schema, id).toSeq.flatMap { field =>
13281328
history.flatMap(fieldById(_, id)).flatMap { older =>
1329-
childFieldsChangedFrom(field, older, fieldName(field))
1329+
structsWithNoFieldsInCommon(field, older, fieldName(field))
13301330
}
13311331
}
13321332
}.distinct
13331333
}
13341334

1335-
/** The nested fields under `field`, at any depth, that `older` lacks or names differently. */
1336-
private def childFieldsChangedFrom(field: Any, older: Any, path: String): Seq[String] = {
1335+
/** `field` if it is a struct sharing no field with `older`, plus the same for its children. */
1336+
private def structsWithNoFieldsInCommon(field: Any, older: Any, path: String): Seq[String] = {
1337+
val children = childFields(field)
13371338
val olderChildren = childFields(older).map(child => fieldIdOf(child) -> child).toMap
1338-
childFields(field).flatMap { child =>
1339-
val name = fieldName(child)
1340-
val childPath = s"$path.$name"
1341-
olderChildren.get(fieldIdOf(child)) match {
1342-
case Some(olderChild) =>
1343-
val olderName = fieldName(olderChild)
1344-
val renamed =
1345-
if (olderName != name) Seq(s"$childPath (renamed from $olderName)") else Nil
1346-
renamed ++ childFieldsChangedFrom(child, olderChild, childPath)
1347-
case None => Seq(s"$childPath (added)")
1339+
val disjoint = isStruct(field) && children.nonEmpty &&
1340+
!children.exists(child => olderChildren.contains(fieldIdOf(child)))
1341+
val here = if (disjoint) Seq(path) else Nil
1342+
here ++ children.flatMap { child =>
1343+
olderChildren.get(fieldIdOf(child)).toSeq.flatMap { olderChild =>
1344+
structsWithNoFieldsInCommon(child, olderChild, s"$path.${fieldName(child)}")
13481345
}
13491346
}
13501347
}
13511348

1349+
private def isStruct(field: Any): Boolean = {
1350+
val fieldType = getMethod(field.getClass, "type").invoke(field)
1351+
getMethod(fieldType.getClass, "isStructType").invoke(fieldType).asInstanceOf[Boolean]
1352+
}
1353+
13521354
/** A struct's fields, a list's element, or a map's key and value. Empty for other types. */
13531355
private def childFields(field: Any): Seq[Any] = {
13541356
import scala.jdk.CollectionConverters._

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -831,16 +831,17 @@ case class CometScanRule(session: SparkSession)
831831
false
832832
}
833833

834-
// Neither iceberg-rust nor the batch adaptation after it matches a data file's nested
835-
// fields to the table's by field id (apache/iceberg-rust#2617). A file written before a
836-
// nested field was added (to a struct, or to a struct inside a list or map) fails the
837-
// native scan with "Incorrect number of arrays for StructArray fields", and a nested field
838-
// renamed since the file was written can read back as NULL. This check walks the schema
839-
// the scan reads (see CometIcebergNativeScanMetadata.taskSchema), which for VERSION AS OF
840-
// can be older than the table's, and also the table and scan schemas, so it can fall back
841-
// when the changed field is pruned away. A FileScanTask does not record which schema wrote
842-
// its file, so fall back when any schema in the table's history lacks a nested field of a
843-
// projected column or names it differently.
834+
// iceberg-rust matches a data file's nested fields to the table's by field id, filling in
835+
// a field the file lacks with NULLs and applying renames, so schema evolution that adds,
836+
// renames, drops or reorders nested fields reads correctly. One shape still differs from
837+
// Spark: a struct (at any depth) whose fields in the schema the scan reads were all added
838+
// after a file was written. The native read returns a struct of NULLs or NULL there,
839+
// depending on what else the scan reads, where Spark returns NULL for a top-level struct
840+
// and a struct of NULLs for a nested one. The schema the scan reads is the task schema (see
841+
// CometIcebergNativeScanMetadata.taskSchema), which holds only the nested fields Spark
842+
// keeps while nested schema pruning is on, and which for VERSION AS OF can be older than
843+
// the table's. A FileScanTask does not record which schema wrote its file, so fall back
844+
// when any schema in the table's history has such a struct in a projected column.
844845
val nestedFieldsSupported =
845846
try {
846847
val schemas = Seq(metadata.tableSchema, metadata.scanSchema) ++ metadata.taskSchema
@@ -850,18 +851,17 @@ case class CometScanRule(session: SparkSession)
850851
} else {
851852
resolvedProjectedFieldIds
852853
}
853-
val changed = schemas
854-
.flatMap { schema =>
855-
IcebergReflection.nestedFieldsAddedOrRenamed(metadata.table, schema, fieldIds)
856-
}
854+
val disjoint = metadata.taskSchema.toSeq
855+
.flatMap(IcebergReflection.structsWithNoFieldsInCommon(metadata.table, _, fieldIds))
857856
.distinct
858857
.sorted
859-
if (changed.nonEmpty) {
860-
fallbackReasons += "Nested fields added or renamed by Iceberg schema evolution are " +
861-
"not yet supported by Comet's native reader, which cannot match them to data " +
862-
s"files written before the change: ${changed.mkString(", ")}"
858+
if (disjoint.nonEmpty) {
859+
fallbackReasons += "Structs whose fields were all added by Iceberg schema " +
860+
"evolution are not yet supported by Comet's native reader, which reads data " +
861+
"files written before the change differently from Spark: " +
862+
disjoint.mkString(", ")
863863
}
864-
changed.isEmpty
864+
disjoint.isEmpty
865865
} catch {
866866
case e: Exception =>
867867
fallbackReasons += "Iceberg reflection failure: could not compare nested fields " +

0 commit comments

Comments
 (0)