Skip to content

Commit 170e5eb

Browse files
committed
Fix the native Iceberg task schema at planning
`taskSchema` read the nested schema pruning config every time it was called: during planning, where CometScanRule checks the schema, and again when the scan ran and serialized it. A query planned with pruning on and run with pruning off, such as a cached DataFrame, read the full schema, which the nested-evolution check never walked. Make it a val, built with the metadata during planning. The struct fallback test plans a VERSION AS OF read with pruning on and runs it with pruning off.
1 parent 86eaa60 commit 170e5eb

2 files changed

Lines changed: 12 additions & 3 deletions

File tree

‎spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2450,10 +2450,12 @@ case class CometIcebergNativeScanMetadata(
24502450

24512451
/**
24522452
* The schema every task of the scan reads with: the pruned task schema while nested schema
2453-
* pruning is on and can apply, the full one otherwise. CometScanRule checks this schema and
2454-
* CometIcebergNativeScan serializes it, so the two cannot disagree. None falls back to Spark.
2453+
* pruning is on and can apply, the full one otherwise. It is fixed when `extract` builds the
2454+
* metadata during planning, so the schema CometScanRule checks is the one
2455+
* CometIcebergNativeScan serializes at execution, even if the config changes in between. None
2456+
* falls back to Spark.
24552457
*/
2456-
def taskSchema: Option[Any] =
2458+
val taskSchema: Option[Any] =
24572459
prunedTaskSchema
24582460
.filter(_ => CometConf.COMET_ICEBERG_NESTED_SCHEMA_PRUNING_ENABLED.get())
24592461
.orElse(fullTaskSchema)

‎spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1880,6 +1880,13 @@ class CometIcebergNativeSuite
18801880
withSQLConf(CometConf.COMET_ICEBERG_NESTED_SCHEMA_PRUNING_ENABLED.key -> "false") {
18811881
checkNestedFieldEvolutionFallback(unpartitionedQuery, "s.b (added)")
18821882
}
1883+
// The task schema is chosen during planning, so a query planned with pruning on still
1884+
// reads the pruned schema when it runs with pruning off.
1885+
val planned = spark.sql(unpartitionedQuery)
1886+
assertSingleNativeScan(planned.queryExecution.executedPlan)
1887+
withSQLConf(CometConf.COMET_ICEBERG_NESTED_SCHEMA_PRUNING_ENABLED.key -> "false") {
1888+
checkAnswer(planned, Seq(Row(1, 100, 10), Row(2, 200, 20)))
1889+
}
18831890

18841891
Seq(table, historical, unpartitioned).foreach(t => spark.sql(s"DROP TABLE $t"))
18851892
}

0 commit comments

Comments
 (0)