Skip to content

Commit 5ee8d8f

Browse files
committed
review: tighten the CometRule docstring, test and docs
The docstring claimed operator conversion against unconverted Spark scans converts nothing. That is wrong: with spark.comet.convert.parquet.enabled=true CometExecRule bridges the scan with a CometSparkToColumnarExec and converts the operators above it, and it converts local table scans, empty relations and shuffles with no scan conversion at all. State the narrower property instead, drop the archaeology about what this PR changed, and record how the pass is named in Spark's plan change log. The test named an invariant that only held because spark.comet.convert.parquet.enabled defaults to false, and it never ran CometRule. Pin the conf explicitly and assert both directions: CometExecRule alone leaves the FileSourceScanExec in place, CometRule on the same query produces a CometNativeScanExec. Each direction needs its own plan, because fallback reasons are node tags and CometNativeScan.isSupported declines a scan already carrying one. Also switch RevertNativeForTransitionHeavyStagesSuite.applyFullColumnarPipeline, which chained the two rules by hand, to CometRule, and update plugin_overview.md to describe the single registered rule and its two phases.
1 parent 8dd66d4 commit 5ee8d8f

4 files changed

Lines changed: 47 additions & 31 deletions

File tree

‎docs/source/contributor-guide/plugin_overview.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -47,11 +47,15 @@ The plugin also registers `CometSparkSessionExtensions` with Spark's extension A
4747

4848
## CometSparkSessionExtensions
4949

50-
On initialization, this class registers two physical plan optimization rules with Spark: `CometScanRule`
51-
and `CometExecRule`. These rules run whenever a query stage is being planned during Adaptive Query Execution, and
52-
run once for the entire plan when Adaptive Query Execution is disabled.
50+
On initialization, this class registers one physical plan optimization rule with Spark: `CometRule`. It runs whenever
51+
a query stage is being planned during Adaptive Query Execution, and runs once for the entire plan when Adaptive Query
52+
Execution is disabled.
5353

54-
### CometScanRule
54+
`CometRule` is two phases, applied in order: scan conversion (`CometScanRule`), then operator conversion
55+
(`CometExecRule`). The order matters, because operator conversion builds its native plan up from the nodes that scan
56+
conversion produces. Each phase is described below.
57+
58+
### Phase 1: CometScanRule
5559

5660
`CometScanRule` replaces any Parquet scans with Comet operators. There are different paths for Spark v1 and v2 data sources.
5761

@@ -68,7 +72,7 @@ convert the output from Spark's scan to Arrow arrays. Note that both `spark.come
6872
Refer to the [Supported Spark Data Types](https://datafusion.apache.org/comet/user-guide/datatypes.html) section
6973
in the contributor guide to see a list of currently supported data types.
7074

71-
### CometExecRule
75+
### Phase 2: CometExecRule
7276

7377
This rule traverses bottom-up from the original Spark plan and attempts to replace each operator with a Comet equivalent.
7478
For example, a `ProjectExec` will be replaced by `CometProjectExec`.

‎spark/src/main/scala/org/apache/comet/rules/CometRule.scala‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,17 +26,16 @@ import org.apache.spark.sql.execution.SparkPlan
2626
/**
2727
* Comet's plan conversion pass: scan conversion followed by operator conversion.
2828
*
29-
* The two were previously registered as separate rules, adjacently, in both the columnar and the
30-
* query-stage-prep paths. Nothing ever ran between them, and neither is useful on its own -
31-
* [[CometExecRule]] seeds its native chain only from the nodes [[CometScanRule]] produces
32-
* (`CometScanExec`, `CometBatchScanExec`, `CometContribScanMarker`), so operator conversion
33-
* against unconverted Spark scans converts nothing. Composing them here makes that ordering an
34-
* invariant of the code rather than of the registration order, and gives callers that need the
35-
* whole conversion - rather than half of it - a single entry point.
29+
* Native scans come only from the nodes [[CometScanRule]] produces (`CometScanExec`,
30+
* `CometBatchScanExec`, `CometContribScanMarker`), so [[CometExecRule]] must run after it.
31+
* Running [[CometExecRule]] alone leaves scans on Spark's readers. Composing the two here makes
32+
* that ordering part of the code instead of the order the rules are registered in, and gives
33+
* callers that need the whole conversion a single entry point.
3634
*
37-
* The two rules keep their own classes, files and tests; this only fixes how they are sequenced.
38-
* `ruleName` in `spark.comet.explain.transformations` output is still each inner rule's own,
39-
* since this delegates to their `apply`.
35+
* `spark.comet.explain.transformations` logs each inner rule under its own `ruleName`, since this
36+
* delegates to their `apply`. Spark's own plan change log sees one rule: query-stage preparation
37+
* logs this pass as `org.apache.comet.rules.CometRule`, which is the name
38+
* `spark.sql.planChangeLog.rules` has to match.
4039
*/
4140
case class CometRule(session: SparkSession) extends Rule[SparkPlan] {
4241

‎spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala‎

Lines changed: 26 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1086,27 +1086,40 @@ class CometExecRuleSuite extends CometTestBase {
10861086
}
10871087
}
10881088

1089-
test("operator conversion alone converts nothing without scan conversion") {
1089+
test("scan conversion must run before operator conversion") {
10901090
withTempPath { path =>
10911091
createTestDataFrame.write.parquet(path.toString)
10921092
withTempView("test_data") {
10931093
spark.read.parquet(path.toString).createOrReplaceTempView("test_data")
1094-
val sparkPlan =
1095-
createSparkPlan(spark, "SELECT id, id * 2 as doubled FROM test_data WHERE id % 2 == 0")
1096-
assert(countOperators(sparkPlan, classOf[FileSourceScanExec]) == 1)
1094+
val query = "SELECT id, id * 2 as doubled FROM test_data WHERE id % 2 == 0"
1095+
1096+
// One plan per rule application. Fallback reasons are recorded as tags on the Spark
1097+
// nodes, and CometNativeScan.isSupported declines a scan already carrying one, so
1098+
// reusing the plan the exec rule just refused would hold the second case down.
1099+
val forExecRule = stripAQEPlan(createSparkPlan(spark, query))
1100+
val forCometRule = stripAQEPlan(createSparkPlan(spark, query))
1101+
assert(countOperators(forExecRule, classOf[FileSourceScanExec]) == 1)
1102+
assert(countOperators(forCometRule, classOf[FileSourceScanExec]) == 1)
10971103

1098-
// CometExecRule seeds its native chain only from the nodes CometScanRule produces, so on
1099-
// its own it converts nothing: the scan is untouched and every operator above it is
1100-
// refused for want of Arrow input. This is why the two are composed into `CometRule`
1101-
// rather than registered as independent rules that happen to run in the right order.
11021104
withSQLConf(
11031105
CometConf.COMET_ENABLED.key -> "true",
1104-
CometConf.COMET_EXEC_ENABLED.key -> "true") {
1105-
val execOnly = CometExecRule(spark).apply(stripAQEPlan(sparkPlan))
1106-
assert(countOperators(execOnly, classOf[FileSourceScanExec]) == 1)
1106+
CometConf.COMET_EXEC_ENABLED.key -> "true",
1107+
// Off by default, but pinned here: with it on, CometExecRule bridges the unconverted
1108+
// scan with a CometSparkToColumnarExec and converts the operators above it, which is a
1109+
// different path from the one under test.
1110+
CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.key -> "false") {
1111+
// CometExecRule builds its native plan up from the nodes CometScanRule produces, so on
1112+
// its own it leaves the scan on Spark's reader.
1113+
assert(
1114+
countOperators(
1115+
CometExecRule(spark).apply(forExecRule),
1116+
classOf[FileSourceScanExec]) == 1)
1117+
// CometRule runs both phases, in that order. This fails if the scan phase is ever
1118+
// reordered or dropped.
11071119
assert(
1108-
stripAQEPlan(execOnly).collect { case p: CometNativeExec => p }.isEmpty,
1109-
s"operator conversion alone should convert nothing, got:\n$execOnly")
1120+
countOperators(
1121+
CometRule(spark).apply(forCometRule),
1122+
classOf[CometNativeScanExec]) == 1)
11101123
}
11111124
}
11121125
}

‎spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,9 @@ class RevertNativeForTransitionHeavyStagesSuite extends CometTestBase {
4444
}
4545

4646
private def applyFullColumnarPipeline(plan: SparkPlan): SparkPlan = {
47-
val cometPlan = CometScanRule(spark).apply(plan)
48-
val execPlan = CometExecRule(spark).apply(cometPlan)
49-
val withTransitions = ApplyColumnarRulesAndInsertTransitions(Seq.empty, false).apply(execPlan)
47+
val cometPlan = CometRule(spark).apply(plan)
48+
val withTransitions =
49+
ApplyColumnarRulesAndInsertTransitions(Seq.empty, false).apply(cometPlan)
5050
EliminateRedundantTransitions(spark).apply(withTransitions)
5151
}
5252

0 commit comments

Comments
 (0)