Skip to content

Commit 2f94707

Browse files
committed
Address review: report the outer query and include post-columnar rules
Plan-only reporting keyed its once-per-query slot on the SQL execution ID alone. Spark prepares scalar and DPP subqueries as their own top-level plans before the outer plan reaches the conversion rules, so a nested subquery took the slot and the outer query - the plan being evaluated - was never reported. Reports are now keyed on the execution ID and the plan, so the outer query always gets one, while AQE's per-stage and per-re-optimization applications of the rule are recognised as re-plans and stay quiet. The preview also stopped at operator conversion, before Spark inserts columnar transitions and Comet runs its post-columnar rules. It therefore counted operators that RevertNativeForTransitionHeavyStages removes and counted transitions before they existed. The preview now inserts transitions and applies both post-columnar rules, using a new applyToAllStages entry point so that every shuffle boundary is visited rather than only the topmost stage. Tests capture the logged report and assert that the outer query appears under both AQE modes, and that the reported coverage matches the coverage of the plan Comet really executes when stage reversion fires.
1 parent eb514de commit 2f94707

5 files changed

Lines changed: 229 additions & 23 deletions

File tree

‎docs/source/user-guide/latest/understanding-comet-plans.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -218,14 +218,30 @@ execution.
218218

219219
The log line is prefixed with `[Comet plan-only]` and includes the same
220220
annotated plan and summary as `spark.comet.explain.format=verbose` produces
221-
against a normal Comet plan. Under AQE the report is emitted once per SQL
222-
execution.
221+
against a normal Comet plan. The preview goes through the whole Comet planning
222+
sequence, not just operator conversion: Spark's columnar transitions are
223+
inserted and Comet's post-columnar rules
224+
(`RevertNativeForTransitionHeavyStages`, `EliminateRedundantTransitions`) are
225+
applied, so a stage that Comet would have reverted to Spark for having too many
226+
transitions is reported as reverted.
227+
228+
Spark prepares some plans on their own, ahead of the query that contains them —
229+
scalar subqueries and dynamic partition pruning subqueries, for instance — so a
230+
query gets one report per independently planned plan: one for the outer query,
231+
plus one per such subquery. Repeat applications of the same plan are not
232+
reported again: under AQE, neither the per-stage applications nor the
233+
applications that follow each adaptive re-optimization add reports.
223234

224235
The estimate reflects Scala-side conversion only. The native plan is never
225236
handed to DataFusion, so anything that would have failed in DataFusion's
226237
`create_plan` still counts as accelerated. Treat the percentage as an upper
227238
bound.
228239

240+
Under AQE there is a second reason to treat the report as an estimate: it
241+
describes the plan as it stands before any adaptive re-planning, and the
242+
post-columnar rules are applied to that whole plan at once rather than to each
243+
stage as it is created. Coverage of the plan AQE finally executes can differ.
244+
229245
The config requires `spark.comet.exec.enabled=true`. With Comet exec disabled
230246
the rule that emits the report does not run.
231247

‎spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,9 @@ class CometSparkSessionExtensions
9797
// No-op on Spark 3.5+; see CometSpark34AqeDppFallbackRule's class docstring.
9898
injectPreSpark35QueryStagePrepRuleShim(extensions, CometSpark34AqeDppFallbackRule)
9999
extensions.injectQueryStagePrepRule { session => CometScanRule(session) }
100-
extensions.injectQueryStagePrepRule { session => CometExecRule(session) }
100+
extensions.injectQueryStagePrepRule { session =>
101+
CometExecRule(session, queryStagePrep = true)
102+
}
101103
injectQueryStageOptimizerRuleShim(extensions, CometPlanAdaptiveDynamicPruningFilters)
102104
injectQueryStageOptimizerRuleShim(extensions, CometReuseSubquery)
103105
extensions.injectPlannerStrategy { session => IcebergWriteStrategy(session) }
@@ -111,6 +113,8 @@ class CometSparkSessionExtensions
111113
override def preColumnarTransitions: Rule[SparkPlan] = CometExecRule(session)
112114

113115
override def postColumnarTransitions: Rule[SparkPlan] = {
116+
// Keep in sync with `CometExecRule.reportPlanOnlyCoverage`, which replays these rules over
117+
// the plan it previews so that plan-only reports describe the plan that would have run.
114118
val rules =
115119
Seq(RevertNativeForTransitionHeavyStages(session), EliminateRedundantTransitions(session))
116120
plan => rules.foldLeft(plan) { case (p, rule) => rule(p) }

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 90 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ import org.apache.spark.sql.comet._
3232
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
3333
import org.apache.spark.sql.comet.util.Utils
3434
import org.apache.spark.sql.execution._
35-
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, AQEShuffleReadExec, BroadcastQueryStageExec, ShuffleQueryStageExec}
35+
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, AQEShuffleReadExec, BroadcastQueryStageExec, QueryStageExec, ShuffleQueryStageExec}
3636
import org.apache.spark.sql.execution.aggregate.{BaseAggregateExec, HashAggregateExec, ObjectHashAggregateExec}
3737
import org.apache.spark.sql.execution.command.{DataWritingCommandExec, ExecutedCommandExec}
3838
import org.apache.spark.sql.execution.datasources.WriteFilesExec
@@ -117,32 +117,89 @@ object CometExecRule {
117117
org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometBroadcast")
118118

119119
/**
120-
* SQL execution IDs for which plan-only mode has already logged its report, as a bounded LRU.
121-
* Dedupes the WARN under AQE, where `CometExecRule` fires once against the full initial plan
122-
* and again per stage. Same synchronized-`LinkedHashMap` LRU pattern used by
123-
* `IcebergPlanDataInjector.commonCache`.
120+
* A bounded set of keys, used for plan-only reporting state. Evicts in LRU order once `limit`
121+
* keys are held, so a long-lived driver retains a fixed amount of reporting state. Same
122+
* synchronized-`LinkedHashMap` pattern used by `IcebergPlanDataInjector.commonCache`.
124123
*/
124+
private class BoundedKeySet(limit: Int) {
125+
private val keys: java.util.Map[String, java.lang.Boolean] =
126+
java.util.Collections.synchronizedMap(
127+
new java.util.LinkedHashMap[String, java.lang.Boolean](16, 0.75f, true) {
128+
override def removeEldestEntry(
129+
eldest: java.util.Map.Entry[String, java.lang.Boolean]): Boolean = size() > limit
130+
})
131+
132+
/** Adds `key`, returning true if it was not already present. */
133+
def add(key: String): Boolean = keys.put(key, java.lang.Boolean.TRUE) == null
134+
135+
def contains(key: String): Boolean = keys.containsKey(key)
136+
}
137+
125138
private val PLAN_ONLY_REPORTED_LIMIT = 1024
126-
private val planOnlyReportedIds: java.util.Map[String, java.lang.Boolean] =
127-
java.util.Collections.synchronizedMap(
128-
new java.util.LinkedHashMap[String, java.lang.Boolean](16, 0.75f, false) {
129-
override def removeEldestEntry(
130-
eldest: java.util.Map.Entry[String, java.lang.Boolean]): Boolean =
131-
size() > PLAN_ONLY_REPORTED_LIMIT
132-
})
133-
134-
private[comet] def markPlanOnlyReported(executionId: Option[String]): Boolean = {
139+
140+
/** `executionId:planFingerprint` keys that plan-only mode has already reported. */
141+
private val planOnlyReportedPlans = new BoundedKeySet(PLAN_ONLY_REPORTED_LIMIT)
142+
143+
/** Execution IDs whose plan-only report came from the query-stage-prep rule. */
144+
private val planOnlyPrepReportedIds = new BoundedKeySet(PLAN_ONLY_REPORTED_LIMIT)
145+
146+
/**
147+
* Whether plan-only mode should report `plan`, recording that it did so.
148+
*
149+
* Spark applies this rule many times during one SQL execution, and only some of those
150+
* applications correspond to a plan the user is asking about:
151+
*
152+
* - Each scalar subquery and DPP subquery is prepared as its own top-level plan, and that
153+
* happens *before* the outer plan reaches the conversion rules. Keying the report on the
154+
* execution ID alone therefore let a nested subquery consume the slot and suppressed the
155+
* outer plan, which is the plan being evaluated. Keying on the execution ID *and* the plan
156+
* gives the outer plan its own report and each separately prepared subquery theirs.
157+
* - Under AQE the rule also runs once per query stage (as a columnar rule) and again on every
158+
* re-optimization (as a query-stage-prep rule). Those are re-planning of a plan already
159+
* reported, so a `plan` containing query stages is skipped, and once the query-stage-prep
160+
* rule has reported an execution the columnar applications for it stay quiet.
161+
*
162+
* @param queryStagePrep
163+
* whether the calling rule instance is registered as a query-stage-prep rule.
164+
*/
165+
private[comet] def shouldReportPlanOnly(
166+
executionId: Option[String],
167+
plan: SparkPlan,
168+
queryStagePrep: Boolean): Boolean = {
135169
executionId match {
136-
case None => true
137-
case Some(id) => planOnlyReportedIds.put(id, java.lang.Boolean.TRUE) == null
170+
case None =>
171+
// No execution ID means the plan is being built outside an action (`df.explain`, or
172+
// reading `queryExecution.executedPlan` directly). AQE creates no stages in that case, so
173+
// there is nothing to dedupe against.
174+
true
175+
case Some(id) =>
176+
if (plan.exists(_.isInstanceOf[QueryStageExec])) {
177+
// An AQE stage plan or a re-optimized plan: a re-plan of what we already reported.
178+
false
179+
} else {
180+
if (queryStagePrep) {
181+
planOnlyPrepReportedIds.add(id)
182+
} else if (planOnlyPrepReportedIds.contains(id)) {
183+
return false
184+
}
185+
// The plan's structural hash identifies it: node tags are not part of it, so the same
186+
// plan applied twice keys the same and is reported once.
187+
planOnlyReportedPlans.add(s"$id:${plan.hashCode()}")
188+
}
138189
}
139190
}
140191
}
141192

142193
/**
143194
* Spark physical optimizer rule for replacing Spark operators with Comet operators.
195+
*
196+
* @param queryStagePrep
197+
* true for the instance registered with `injectQueryStagePrepRule`, which under AQE sees the
198+
* whole initial plan, and false for the one registered as a columnar rule, which under AQE sees
199+
* one query stage at a time. Only plan-only reporting reads this; see
200+
* [[CometExecRule.shouldReportPlanOnly]].
144201
*/
145-
case class CometExecRule(session: SparkSession)
202+
case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
146203
extends Rule[SparkPlan]
147204
with ShimSubqueryBroadcast {
148205

@@ -599,9 +656,23 @@ case class CometExecRule(session: SparkSession)
599656
* Build the Comet plan we would have executed and log it. Called from `_apply` in plan-only
600657
* mode; the built plan is discarded. Passes `forPreview = true` through the nested calls so
601658
* both rules run their normal transforms instead of short-circuiting.
659+
*
660+
* Conversion is only the first half of Comet planning. Normally Spark then inserts the columnar
661+
* transitions and runs Comet's post-columnar rules (see
662+
* `CometSparkSessionExtensions.CometExecColumnar.postColumnarTransitions`), which can revert
663+
* whole stages back to Spark and drop redundant transitions. Those steps run here too, so the
664+
* report describes the plan that would really have executed and counts the transitions that
665+
* would really have been there, rather than the pre-transition conversion result.
666+
*
667+
* `RevertNativeForTransitionHeavyStages` is applied with `applyToAllStages` because the preview
668+
* holds the whole plan at once, whereas under AQE Spark hands that rule one stage at a time.
602669
*/
603670
private def reportPlanOnlyCoverage(plan: SparkPlan): Unit = {
604-
val preview = _apply(CometScanRule(session)._apply(plan), forPreview = true)
671+
val converted = _apply(CometScanRule(session)._apply(plan), forPreview = true)
672+
val withTransitions =
673+
ApplyColumnarRulesAndInsertTransitions(Seq.empty, outputsColumnar = false).apply(converted)
674+
val reverted = RevertNativeForTransitionHeavyStages(session).applyToAllStages(withTransitions)
675+
val preview = EliminateRedundantTransitions(session).apply(reverted)
605676
logWarning(s"[Comet plan-only]\n${new ExtendedExplainInfo().generateExtendedInfo(preview)}")
606677
}
607678

@@ -629,7 +700,7 @@ case class CometExecRule(session: SparkSession)
629700
if (!forPreview && CometConf.COMET_EXPLAIN_PLAN_ONLY_ENABLED.get()) {
630701
val executionId = Option(
631702
session.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY))
632-
if (CometExecRule.markPlanOnlyReported(executionId)) {
703+
if (CometExecRule.shouldReportPlanOnly(executionId, plan, queryStagePrep)) {
633704
reportPlanOnlyCoverage(plan)
634705
}
635706
return plan

‎spark/src/main/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStages.scala‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,19 @@ case class RevertNativeForTransitionHeavyStages(session: SparkSession)
7676
.getOrElse(withRevertedStages)
7777
}
7878

79+
/**
80+
* Applies the revert decision to every stage of `plan`, regardless of whether AQE is enabled.
81+
*
82+
* `apply` picks the AQE branch when AQE is on because Spark hands it a single query stage at a
83+
* time there, so only the topmost stage of `plan` is considered. Callers holding a whole plan
84+
* that has not been split into stages - the plan-only preview in
85+
* `CometExecRule.reportPlanOnlyCoverage` - need every shuffle boundary visited to see the
86+
* reversions that the real per-stage applications would make.
87+
*/
88+
private[rules] def applyToAllStages(plan: SparkPlan): SparkPlan = {
89+
if (!enabled) plan else applyForNonAQE(plan)
90+
}
91+
7992
/**
8093
* Reverts the stage if C2R count exceeds threshold. Wraps in R2C if exchange needs columnar.
8194
*/

‎spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala‎

Lines changed: 103 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ package org.apache.comet.rules
2121

2222
import scala.util.Random
2323

24+
import org.apache.logging.log4j.Level
2425
import org.apache.spark.sql._
2526
import org.apache.spark.sql.catalyst.FunctionIdentifier
2627
import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionInfo}
@@ -34,7 +35,7 @@ import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, ShuffleEx
3435
import org.apache.spark.sql.internal.SQLConf
3536
import org.apache.spark.sql.types.{DataTypes, StructField, StructType}
3637

37-
import org.apache.comet.{CometConf, CometExplainInfo}
38+
import org.apache.comet.{CometConf, CometCoverageStats, CometExplainInfo}
3839
import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark42Plus}
3940
import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator}
4041

@@ -857,4 +858,105 @@ class CometExecRuleSuite extends CometTestBase {
857858
}
858859
}
859860

861+
private val PLAN_ONLY_PREFIX = "[Comet plan-only]"
862+
863+
/** Runs `f` and returns the `[Comet plan-only]` reports that `CometExecRule` logged. */
864+
private def capturePlanOnlyReports(f: => Unit): Seq[String] = {
865+
val appender = new LogAppender("Comet plan-only reports")
866+
withLogAppender(
867+
appender,
868+
loggerNames = Seq(classOf[CometExecRule].getName),
869+
level = Some(Level.WARN)) {
870+
f
871+
}
872+
appender.loggingEvents
873+
.map(_.getMessage.getFormattedMessage)
874+
.filter(_.startsWith(PLAN_ONLY_PREFIX))
875+
.toSeq
876+
}
877+
878+
/** The `Comet accelerated N out of M eligible operators` counts in a plan-only report. */
879+
private def coverageOf(report: String): (Int, Int) = {
880+
val pattern = """Comet accelerated (\d+) out of (\d+) eligible operators""".r
881+
pattern
882+
.findFirstMatchIn(report)
883+
.map(m => (m.group(1).toInt, m.group(2).toInt))
884+
.getOrElse(fail(s"report has no coverage summary:\n$report"))
885+
}
886+
887+
// The outer query is planned after any subquery it contains, so a report slot owned by the
888+
// first plan Spark prepares would describe the subquery and never the query being evaluated.
889+
for (aqe <- Seq(true, false)) {
890+
test(s"plan-only mode: report describes the outer query, not just a subquery (AQE=$aqe)") {
891+
withSQLConf(
892+
SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
893+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString,
894+
CometConf.COMET_ENABLED.key -> "true",
895+
CometConf.COMET_EXEC_ENABLED.key -> "true",
896+
CometConf.COMET_EXPLAIN_PLAN_ONLY_ENABLED.key -> "true") {
897+
withParquetTable((0 until 100).map(i => (i, i % 5)), "tbl") {
898+
val reports = capturePlanOnlyReports {
899+
spark.sql("SELECT _1 FROM tbl WHERE _1 > (SELECT max(_2) FROM tbl)").collect()
900+
}
901+
assert(reports.nonEmpty, "expected a plan-only report")
902+
// The outer plan's Filter appears in no subquery plan, so its presence proves the
903+
// outer query was reported and not suppressed by the subquery's earlier planning.
904+
assert(
905+
reports.exists(_.contains("Filter")),
906+
s"no report describes the outer query:\n${reports.mkString("\n\n")}")
907+
assert(
908+
reports.distinct.size == reports.size,
909+
s"the same plan was reported more than once:\n${reports.mkString("\n\n")}")
910+
// Expected: one report for the subquery plan, one for the outer plan. AQE applies the
911+
// rule again per stage and per re-optimization; those must not add reports.
912+
assert(
913+
reports.size <= 4,
914+
s"expected a report per planned plan, got ${reports.size}:\n" +
915+
reports.mkString("\n\n"))
916+
}
917+
}
918+
}
919+
}
920+
921+
test("plan-only mode: coverage accounts for post-columnar stage reversion") {
922+
withSQLConf(
923+
SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
924+
// AQE off so that Spark applies the post-columnar rules to the whole plan exactly once,
925+
// which is what the preview does, making the two directly comparable.
926+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
927+
CometConf.COMET_ENABLED.key -> "true",
928+
CometConf.COMET_EXEC_ENABLED.key -> "true",
929+
CometConf.COMET_EXEC_PROJECT_ENABLED.key -> "false") {
930+
withParquetTable((0 until 100).map(i => (i, i % 5)), "tbl") {
931+
val query = "SELECT _2, count(*), sum(_1) FROM tbl GROUP BY _2"
932+
933+
// Comet accelerates part of this plan when the stage is left alone, so a preview that
934+
// stopped before the post-columnar rules would report a non-zero count below.
935+
withSQLConf(CometConf.COMET_EXEC_TRANSITION_REVERT_ENABLED.key -> "false") {
936+
val df = sql(query)
937+
df.collect()
938+
assert(CometCoverageStats.forPlan(df.queryExecution.executedPlan).cometOperators > 0)
939+
}
940+
941+
withSQLConf(
942+
CometConf.COMET_EXEC_TRANSITION_REVERT_ENABLED.key -> "true",
943+
CometConf.COMET_EXEC_TRANSITION_REVERT_MAX_TRANSITIONS.key -> "0") {
944+
// What Comet really executes with reversion enabled.
945+
val df = sql(query)
946+
df.collect()
947+
val executed = CometCoverageStats.forPlan(df.queryExecution.executedPlan)
948+
949+
val reports = withSQLConf(CometConf.COMET_EXPLAIN_PLAN_ONLY_ENABLED.key -> "true") {
950+
capturePlanOnlyReports(sql(query).collect())
951+
}
952+
assert(reports.size == 1, s"expected one report, got:\n${reports.mkString("\n\n")}")
953+
assert(
954+
coverageOf(reports.head) ==
955+
(executed.cometOperators, executed.cometOperators + executed.sparkOperators),
956+
s"report disagrees with the executed plan ($executed):\n${reports.head}")
957+
}
958+
}
959+
}
960+
}
961+
860962
}

0 commit comments

Comments
 (0)