Skip to content

Commit 518584e

Browse files
authored
fix(iceberg): don't push transform residuals as their source column, fail on residual errors (apache#6154)
icebergExprToProto read a residual predicate's column name with term.ref().name(). On an UnboundTransform, ref() is the transform's source column, so bucket(4, id) = 2 reached iceberg-rust as id = 2, and iceberg-rust filters rows by the pushed predicate. CometScanRule falls back for a bare transform predicate but not for one under AND, OR or NOT, so pairing an Iceberg system-function filter with any other predicate returned too few rows. Decline any term that is not a NamedReference. Residual conversion failures were caught and dropped both inside the converter and at its call site, so a residual the serde declines on purpose and one it failed to read looked the same. Let the converter's exceptions propagate, and fail the query in a new serializeResidual, as serde already does for other reflection failures once CometScanRule has committed the scan. Closes apache#5992.
1 parent 092b6a6 commit 518584e

5 files changed

Lines changed: 281 additions & 71 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -487,6 +487,7 @@ jobs:
487487
org.apache.comet.CometIcebergWriteActionSuite
488488
org.apache.comet.CometIcebergWriteDetectionSuite
489489
org.apache.comet.CometIcebergSystemFunctionSuite
490+
org.apache.comet.CometIcebergResidualPushdownSuite
490491
org.apache.comet.iceberg.IcebergReflectionSuite
491492
org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite
492493
org.apache.comet.csv.CometCsvNativeReadSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,7 @@ jobs:
135135
org.apache.comet.CometIcebergWriteActionSuite
136136
org.apache.comet.CometIcebergWriteDetectionSuite
137137
org.apache.comet.CometIcebergSystemFunctionSuite
138+
org.apache.comet.CometIcebergResidualPushdownSuite
138139
org.apache.comet.iceberg.IcebergReflectionSuite
139140
org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite
140141
org.apache.comet.csv.CometCsvNativeReadSuite

‎spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala‎

Lines changed: 97 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,9 @@ object CometIcebergNativeScan extends CometOperatorSerde[CometBatchScanExec] wit
7575
val AND = "And"
7676
val OR = "Or"
7777
val NOT = "Not"
78+
// The only UnboundTerm shape this serde converts. The others, UnboundTransform and (on
79+
// Iceberg versions that have it) UnboundExtract, are declined; see icebergExprToProto.
80+
val NAMED_REFERENCE = "NamedReference"
7881
}
7982
}
8083

@@ -643,31 +646,48 @@ object CometIcebergNativeScan extends CometOperatorSerde[CometBatchScanExec] wit
643646
}
644647

645648
/**
646-
* Converts an Iceberg residual Expression into an IcebergPredicate for native row-group
647-
* pruning.
649+
* Converts an Iceberg residual Expression into an IcebergPredicate for the native scan.
648650
*
649651
* Residuals come from Iceberg's ResidualEvaluator (partial evaluation of the scan filter
650-
* against each file's partition data). This is only a pruning hint: the CometFilter above the
651-
* scan enforces correctness, so any node or literal we cannot represent yields None (no
652-
* pushdown). Predicates over `pageIndexUnsupportedColumns` also yield None (iceberg-rust cannot
653-
* use those columns in the page index). Uses reflection because Iceberg's expression classes
652+
* against each file's partition data). iceberg-rust prunes row groups and pages with the
653+
* predicate and also filters rows by it, so what this returns may be weaker than the residual
654+
* but never stronger. Iceberg keeps every predicate that can survive into a residual in the
655+
* scan's postScanFilters, so the filter above the scan re-applies whatever is not pushed, but
656+
* it cannot restore rows the scan dropped. Uses reflection because Iceberg's expression classes
654657
* are not on Spark's classpath at planning time; residuals are unbound predicates carrying a
655658
* NamedReference (column name) and a literal.
659+
*
660+
* Not pushing a residual therefore takes one of two forms, kept deliberately distinct:
661+
*
662+
* - `None` is a decision. Every point that returns it has established that this serde does
663+
* not model the node, term shape, operation, column or literal in front of it, so the
664+
* predicate is left to the post-scan filter.
665+
* - An exception means reflection over Iceberg's expression API did something unexpected, so
666+
* no such decision was reached. It propagates instead of folding into `None`, which would
667+
* make the two indistinguishable; see [[serializeResidual]].
656668
*/
657669
def icebergExprToProto(
658670
icebergExpr: Any,
659671
output: Seq[Attribute],
660672
pageIndexUnsupportedColumns: Set[String]): Option[OperatorOuterClass.IcebergPredicate] = {
661-
try {
662-
val exprClass = icebergExpr.getClass
663-
val attributeMap = output.map(attr => attr.name -> attr).toMap
664-
665-
if (exprClass.getName.endsWith(Constants.ExpressionTypes.UNBOUND_PREDICATE)) {
666-
val operation = IcebergReflection.getMethod(exprClass, "op").invoke(icebergExpr).toString
667-
val term = IcebergReflection.getMethod(exprClass, "term").invoke(icebergExpr)
668-
val ref = IcebergReflection.getMethod(term.getClass, "ref").invoke(term)
673+
val exprClass = icebergExpr.getClass
674+
val attributeMap = output.map(attr => attr.name -> attr).toMap
675+
676+
if (exprClass.getName.endsWith(Constants.ExpressionTypes.UNBOUND_PREDICATE)) {
677+
val operation = IcebergReflection.getMethod(exprClass, "op").invoke(icebergExpr).toString
678+
val term = IcebergReflection.getMethod(exprClass, "term").invoke(icebergExpr)
679+
680+
// Only a bare column reference converts. UnboundTransform (and UnboundExtract, on Iceberg
681+
// versions that have it) answers ref() with its *source* column, so reading the name off
682+
// it would push `bucket(4, id) = 2` as `id = 2` and drop every matching row whose id is
683+
// not 2. CometScanRule declines a non-identity transform at planning time, but only when
684+
// the residual is a bare predicate rather than one under AND/OR/NOT, so the term shape is
685+
// checked here too rather than assumed.
686+
if (!term.getClass.getName.endsWith(Constants.ExpressionTypes.NAMED_REFERENCE)) {
687+
None
688+
} else {
669689
val columnName =
670-
IcebergReflection.getMethod(ref.getClass, "name").invoke(ref).asInstanceOf[String]
690+
IcebergReflection.getMethod(term.getClass, "name").invoke(term).asInstanceOf[String]
671691

672692
// Iceberg names a nested reference by its dotted path ("struct.field"), which never matches
673693
// a top-level scan output attribute, so a residual on a nested field drops here. That miss
@@ -694,54 +714,68 @@ object CometIcebergNativeScan extends CometOperatorSerde[CometBatchScanExec] wit
694714
}
695715
}
696716
}
697-
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.AND)) {
698-
val left = icebergExprToProto(
699-
IcebergReflection.getMethod(exprClass, "left").invoke(icebergExpr),
700-
output,
701-
pageIndexUnsupportedColumns)
702-
val right = icebergExprToProto(
703-
IcebergReflection.getMethod(exprClass, "right").invoke(icebergExpr),
704-
output,
705-
pageIndexUnsupportedColumns)
706-
(left, right) match {
707-
// Push the residual only if it converts whole. Dropping a conjunct is safe in positive
708-
// position but strengthens the predicate under a NOT (De Morgan), which would wrongly
709-
// prune, and tracking polarity across arbitrary nesting is error prone. So an
710-
// unconvertible conjunct elides the whole residual; the post-scan CometFilter is exact.
711-
case (Some(l), Some(r)) => Some(logicalPredicate(isAnd = true, l, r))
712-
case _ => None
713-
}
714-
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.OR)) {
715-
val left = icebergExprToProto(
716-
IcebergReflection.getMethod(exprClass, "left").invoke(icebergExpr),
717-
output,
718-
pageIndexUnsupportedColumns)
719-
val right = icebergExprToProto(
720-
IcebergReflection.getMethod(exprClass, "right").invoke(icebergExpr),
721-
output,
722-
pageIndexUnsupportedColumns)
723-
// Dropping a disjunct would strengthen the predicate and wrongly prune, so require both.
724-
(left, right) match {
725-
case (Some(l), Some(r)) => Some(logicalPredicate(isAnd = false, l, r))
726-
case _ => None
727-
}
728-
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.NOT)) {
729-
val child = IcebergReflection.getMethod(exprClass, "child").invoke(icebergExpr)
730-
icebergExprToProto(child, output, pageIndexUnsupportedColumns).map(notPredicate)
731-
} else {
732-
None
733717
}
718+
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.AND)) {
719+
val left = icebergExprToProto(
720+
IcebergReflection.getMethod(exprClass, "left").invoke(icebergExpr),
721+
output,
722+
pageIndexUnsupportedColumns)
723+
val right = icebergExprToProto(
724+
IcebergReflection.getMethod(exprClass, "right").invoke(icebergExpr),
725+
output,
726+
pageIndexUnsupportedColumns)
727+
(left, right) match {
728+
// Push the residual only if it converts whole. Dropping a conjunct is safe in positive
729+
// position but strengthens the predicate under a NOT (De Morgan), which would wrongly
730+
// prune, and tracking polarity across arbitrary nesting is error prone. So an
731+
// unconvertible conjunct elides the whole residual; the post-scan CometFilter is exact.
732+
case (Some(l), Some(r)) => Some(logicalPredicate(isAnd = true, l, r))
733+
case _ => None
734+
}
735+
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.OR)) {
736+
val left = icebergExprToProto(
737+
IcebergReflection.getMethod(exprClass, "left").invoke(icebergExpr),
738+
output,
739+
pageIndexUnsupportedColumns)
740+
val right = icebergExprToProto(
741+
IcebergReflection.getMethod(exprClass, "right").invoke(icebergExpr),
742+
output,
743+
pageIndexUnsupportedColumns)
744+
// Dropping a disjunct would strengthen the predicate and wrongly prune, so require both.
745+
(left, right) match {
746+
case (Some(l), Some(r)) => Some(logicalPredicate(isAnd = false, l, r))
747+
case _ => None
748+
}
749+
} else if (exprClass.getName.endsWith(Constants.ExpressionTypes.NOT)) {
750+
val child = IcebergReflection.getMethod(exprClass, "child").invoke(icebergExpr)
751+
icebergExprToProto(child, output, pageIndexUnsupportedColumns).map(notPredicate)
752+
} else {
753+
// Anything else, such as Expressions.alwaysTrue or alwaysFalse, or a node type a future
754+
// Iceberg introduces, carries no pushdown this serde can express.
755+
None
756+
}
757+
}
758+
759+
/**
760+
* Converts a FileScanTask's residual with [[icebergExprToProto]], failing the query if that
761+
* throws. The converter returns None only for a residual it deliberately leaves to the
762+
* post-scan filter, so a failure means this serde has misread Iceberg's expression API. Serde
763+
* runs after CometScanRule committed the scan to native execution, so there is no fallback to
764+
* take.
765+
*/
766+
private[operator] def serializeResidual(
767+
residual: Any,
768+
output: Seq[Attribute],
769+
pageIndexUnsupportedColumns: Set[String]): Option[OperatorOuterClass.IcebergPredicate] =
770+
try {
771+
icebergExprToProto(residual, output, pageIndexUnsupportedColumns)
734772
} catch {
735-
// Reflection over Iceberg's expression classes can fail on an unexpected shape (e.g. an
736-
// Iceberg version change). A residual is only a pruning hint, so skip pushdown rather than
737-
// fail the scan, but log it: a persistent warning here signals a real API drift to fix.
738773
case e: Exception =>
739-
logWarning(
740-
"Skipping Iceberg residual pushdown; could not convert expression: " +
741-
s"${e.getMessage}")
742-
None
774+
val msg = "Iceberg reflection failure: Failed to convert residual expression " +
775+
s"'$residual' from FileScanTask: ${e.getMessage}"
776+
logError(msg)
777+
throw new RuntimeException(msg, e)
743778
}
744-
}
745779

746780
private def unaryPredicate(
747781
column: String,
@@ -1193,18 +1227,10 @@ object CometIcebergNativeScan extends CometOperatorSerde[CometBatchScanExec] wit
11931227
}
11941228

11951229
val residualExprOpt =
1196-
try {
1197-
icebergExprToProto(
1198-
residualMethod.invoke(task),
1199-
output,
1200-
pageIndexUnsupportedColumns)
1201-
} catch {
1202-
case e: Exception =>
1203-
logWarning(
1204-
"Failed to extract residual expression from FileScanTask: " +
1205-
s"${e.getMessage}")
1206-
None
1207-
}
1230+
serializeResidual(
1231+
residualMethod.invoke(task),
1232+
output,
1233+
pageIndexUnsupportedColumns)
12081234

12091235
residualExprOpt.foreach { residual =>
12101236
val residualIdx = residualToPoolIndex.getOrElseUpdate(
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.comet
21+
22+
import org.apache.spark.SparkConf
23+
import org.apache.spark.sql.CometTestBase
24+
import org.apache.spark.sql.comet.CometIcebergNativeScanExec
25+
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
26+
27+
/**
28+
* Residual pushdown for predicates that only reach the Iceberg scan filter when Iceberg's SQL
29+
* extensions are installed.
30+
*
31+
* Spark resolves an Iceberg system function such as `bucket` to a `StaticInvoke`, which it cannot
32+
* translate into a data source predicate. The extensions' `ReplaceStaticInvoke` rule rewrites it
33+
* so Iceberg can push the comparison, which puts an `UnboundTransform` term into each file's
34+
* residual. CometIcebergNativeSuite runs without the extensions, so it never sees one.
35+
*/
36+
class CometIcebergResidualPushdownSuite
37+
extends CometTestBase
38+
with CometIcebergTestBase
39+
with AdaptiveSparkPlanHelper {
40+
41+
override protected def sparkConf: SparkConf =
42+
super.sparkConf.set(
43+
"spark.sql.extensions",
44+
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
45+
46+
// A transform term's ref() is its source column. Pushing `bucket(4, id) = 2` as `id = 2` would
47+
// have iceberg-rust drop every matching row whose id is not 2, and the exact filter above the
48+
// scan cannot restore them. CometScanRule falls back for a bare transform predicate, so every
49+
// query nests the transform under AND, OR or NOT, which is what reaches the native scan.
50+
test("transform predicates in a residual are not pushed as their source column") {
51+
assume(icebergAvailable, "Iceberg not available in classpath")
52+
53+
withTempIcebergDir { warehouseDir =>
54+
withSQLConf(
55+
"spark.sql.catalog.test_cat" -> "org.apache.iceberg.spark.SparkCatalog",
56+
"spark.sql.catalog.test_cat.type" -> "hadoop",
57+
"spark.sql.catalog.test_cat.warehouse" -> warehouseDir.getAbsolutePath,
58+
CometConf.COMET_ENABLED.key -> "true",
59+
CometConf.COMET_EXEC_ENABLED.key -> "true",
60+
CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
61+
62+
// Unpartitioned, so Iceberg can settle none of these predicates from partition values
63+
// and keeps each one whole in the residual.
64+
spark.sql("""
65+
CREATE TABLE test_cat.db.transform_residual (
66+
id INT,
67+
data STRING,
68+
ts TIMESTAMP
69+
) USING iceberg
70+
TBLPROPERTIES ('format-version' = '2')
71+
""")
72+
spark.sql("""
73+
INSERT INTO test_cat.db.transform_residual
74+
SELECT
75+
CAST(id AS INT),
76+
CONCAT('d', id),
77+
CAST(DATE_ADD(DATE '2024-01-01', CAST(id AS INT)) AS TIMESTAMP)
78+
FROM range(100)
79+
""")
80+
81+
val table = "test_cat.db.transform_residual"
82+
Seq(
83+
// The pushed id = 2 would keep one row, and bucket(4, 2) is not 2.
84+
s"SELECT id FROM $table WHERE test_cat.system.bucket(4, id) = 2 AND data > 'a'",
85+
s"SELECT id FROM $table WHERE test_cat.system.bucket(4, id) = 2 OR id = 50",
86+
// The pushed id != 2 would drop the id = 2 row, which this predicate keeps.
87+
s"SELECT id FROM $table WHERE NOT (test_cat.system.bucket(4, id) = 2) AND data > 'a'",
88+
// The pushed data = 'd1' would drop d10 through d19.
89+
s"SELECT id FROM $table WHERE test_cat.system.truncate(2, data) = 'd1' AND id >= 0",
90+
// The pushed literal is a day count, which would be compared to ts as microseconds.
91+
s"SELECT id FROM $table WHERE test_cat.system.days(ts) = DATE '2024-01-05' AND id >= 0")
92+
.foreach { query =>
93+
val (_, cometPlan) = checkSparkAnswer(query)
94+
val scans = collect(cometPlan) { case scan: CometIcebergNativeScanExec => scan }
95+
assert(
96+
scans.length == 1,
97+
s"Expected the residual to reach a native scan, but found ${scans.length} for " +
98+
s"$query. Plan:\n$cometPlan")
99+
}
100+
101+
spark.sql(s"DROP TABLE $table")
102+
}
103+
}
104+
}
105+
}

0 commit comments

Comments
 (0)