Skip to content

Commit 1e23881

Browse files
committed
fix: recognize Dataset.rdd plans whose deserializer Spark removed
The guard for code reading Dataset.rdd looked for the DeserializeToObjectExec that Dataset.rdd puts at the root of the plan. When the Dataset ends in a typed operation such as map, Spark's EliminateSerialization drops that deserializer together with the operation's serializer, so the root is the operation itself, or a typed filter over it. The output of an earlier typed operation was then still converted, and map(f).filter(...).map(g).rdd.take(1) ran f on rows that Spark never reaches. The guard now takes any root that produces objects, under a filter or a project, as code reading Dataset.rdd. A Dataset's own plan ends in rows, so no other plan has such a root.
1 parent bd36c6b commit 1e23881

2 files changed

Lines changed: 21 additions & 7 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1340,13 +1340,18 @@ case class CometExecRule(session: SparkSession)
13401340
}
13411341
op.children.foreach(visit(_, childReader))
13421342
}
1343-
// `Dataset.rdd` plans a `DeserializeToObjectExec` at the root, and the RDD's own code
1344-
// decides how much of it to read, as `take(1)` does.
1345-
val rootReader = plan match {
1346-
case _: DeserializeToObjectExec => Some("code reading Dataset.rdd")
1347-
case _ => None
1343+
// `Dataset.rdd` reads the objects the plan produces, and the RDD's own code decides how many
1344+
// of them to read, as `take(1)` does. The plan's root is the `DeserializeToObjectExec` that
1345+
// `Dataset.rdd` adds. When the Dataset ends in a typed operation such as `map`, Spark's
1346+
// `EliminateSerialization` drops that deserializer together with the operation's serializer,
1347+
// so the root is the operation itself, which produces objects too, or a typed filter or a
1348+
// project over it. A Dataset's own plan ends in rows, so no other plan has such a root.
1349+
def producesObjects(op: SparkPlan): Boolean = op match {
1350+
case _: ObjectProducerExec => true
1351+
case _: FilterExec | _: ProjectExec => producesObjects(op.children.head)
1352+
case _ => false
13481353
}
1349-
visit(plan, rootReader)
1354+
visit(plan, if (producesObjects(plan)) Some("code reading Dataset.rdd") else None)
13501355
}
13511356

13521357
/**

‎spark/src/test/scala/org/apache/comet/exec/CometTypedDatasetSuite.scala‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -317,7 +317,16 @@ class CometTypedDatasetSuite extends CometTestBase {
317317
}
318318

319319
convertTest("code reading Dataset.rdd does not evaluate typed Dataset rows it never reads") {
320-
assert(failsOnRow30.filter(col("value") > 0L).rdd.take(1).toSeq == Seq(1L))
320+
Seq("true", "false").foreach { aqe =>
321+
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe) {
322+
val filtered = failsOnRow30.filter(col("value") > 0L)
323+
assert(filtered.rdd.take(1).toSeq == Seq(1L))
324+
// When the Dataset ends in a typed operation, Spark drops the deserializer that `rdd`
325+
// adds, so the root of the plan is that operation, or a typed filter over it.
326+
assert(filtered.map(_ + 1L).rdd.take(1).toSeq == Seq(2L))
327+
assert(filtered.map(_ + 1L).filter((v: Long) => v > 0L).rdd.take(1).toSeq == Seq(2L))
328+
}
329+
}
321330
}
322331

323332
convertTest("a limit above an aggregate keeps the conversion the aggregate reads all of") {

0 commit comments

Comments
 (0)