Repository navigation
Conversation
uros-b
left a comment
There was a problem hiding this comment.
Overall, the plan shape seems like the right fix (Filter under both Window operators, rank_sum = 1), but I think that this still needs a flag since this is a user-facing result / behaviour change. Please add an internal flag with .withBindingPolicy(ConfigBindingPolicy.SESSION), defaulting to the corrected behaviour, and name it in the migration bullet under “Upgrading from Spark SQL 4.3 to 4.4”. And yeah, a couple of extra goldens would help here indeed (HAVING with an aggregate, generator in the SELECT list etc.)
|
Also untested:
|
uros-b
left a comment
There was a problem hiding this comment.
Most importantly, it seems that generator queries still get wrong results... We should definitely investigate more and add proper tests
|
The AI review requested by @uros-b did not finish. A committer can add the |
|
Thank you for addressing all comments! Let's wait for AI review here as well |
spark-test
left a comment
There was a problem hiding this comment.
Review summary
The nested-window fix holds for plain GROUP BY: addWindowWithHaving places HAVING below every window layer, and the new goldens pin that plan under both values of the legacy flag.
In generator queries, the new ResolveUnresolvedHaving walk can insert the HAVING filter between two Window operators when the condition reads a window alias whose Window has another Window above it. The upper window's values then change silently and depend on SELECT-list order. Before this change, such a filter stayed above every window.
ROLLUP, CUBE and GROUPING SETS queries whose HAVING needs no type coercion still evaluate HAVING after every window layer, whatever the flag is set to, because their HAVING becomes a Filter before either changed rule runs. The migration note and the config doc state the fix without that exception.
The new addWindowWithHaving Scaladoc also describes only the method's default branch.
Review-level concerns
- Shared problem: Where HAVING lands relative to window functions is decided in three separate analyzer routes. When HAVING is still attached to its Aggregate, the new recursive extraction places it. When a generator has rewritten the plan, the final HAVING pass walks the finished plan. For ROLLUP, CUBE and GROUPING SETS, the grouping-analytics rewrite turns HAVING into a Filter before any window is extracted. The change makes the first route correct, gives the second a walk that can stop inside a window chain, and leaves the third as it was, so the same HAVING semantics now depend on which route a query takes. Unlike plain GROUP BY, the generator route and, for conditions that need no coercion, the grouping-analytics route also accept a HAVING condition that reads a SELECT-list window alias, so the fixes for both need one rule: HAVING goes below every window layer, or stays above all of them when it reads a window's output, but never between them. Covering each route with SQL golden tests under both values of spark.sql.legacy.evaluateHavingAfterInnerWindow keeps the routes from drifting apart again.
Findings
3 total: 0 P0, 1 P1, 1 P2, 1 P3.
Blocking (P1)
- HAVING filter can land between two Window operators in generator queries —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4972— see inline.
Non-blocking (P2)
- ROLLUP, CUBE and GROUPING SETS still evaluate HAVING after nested windows —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3866— see inline.
Nit (P3)
- addWindowWithHaving Scaladoc omits the legacy-flag branch —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3864— see inline.
Generated by Omnigent on Databricks
| case window: Window if filter.condition.references.subsetOf(window.child.outputSet) => | ||
| insertFilterBeforeAllWindows(filter.copy(child = window.child), foundWindow = true) | ||
| .map(child => window.withNewChildren(Seq(child))) | ||
| case _ if foundWindow && filter.condition.references.subsetOf(filter.child.outputSet) => |
There was a problem hiding this comment.
Blocking (P1): Once foundWindow is true, this fallback accepts any stop point, including a lower Window of the same chain whose child lacks the condition's references. That breaks the all-or-none contract in the method doc above ("Returns None unless the condition can be evaluated below every Window in the chain").
It is reachable from SQL. In a generator query, HAVING is resolved against the post-generator Project, so it can read a SELECT-list window alias:
SELECT explode(array(k)) AS g,
RANK() OVER (ORDER BY k) AS r,
COUNT(*) OVER () AS c
FROM VALUES ('a', 1), ('b', 2) AS t(k, v)
GROUP BY k
HAVING r = 1addWindow stacks Window[c] over Window[r]. The walk passes Window[c], fails the guard at Window[r] (r is not in the Aggregate output), and this case returns Filter(r = 1, Window[r]). COUNT(*) OVER () therefore counts only the surviving row, and the query returns (a, 1, 1). Before this change, and with the legacy flag set to true, the filter stays above both windows and the query returns (a, 1, 2). Listing c before r brings back c = 2, so the value now depends on SELECT-list order. Neither the migration note nor the config doc describes this, and no test covers it.
Suggested fix: return None whenever the walk stops at a Window, or at a Project or Generate that still has a Window below it, so the filter is inserted only once every Window of the chain is above the stop point.
This extends an existing discussion.
Verification:
- Regression: A generator query whose HAVING references the alias of a window computed below another window spec keeps the HAVING filter above both windows and returns the base-branch values, and the same query with the two window expressions listed in the opposite order returns the same values.
- Behavior: Generator queries with nested windows and HAVING on a grouping key still place the HAVING filter directly above the Aggregate and return rank_sum 1.
- Compatibility: With spark.sql.legacy.evaluateHavingAfterInnerWindow set to true, generator-query plans and results are unchanged.
- Inspection: The default-path walk returns no placement whenever a Window would remain below the inserted filter, matching the documented contract that HAVING goes below every Window in the chain or not at all.
There was a problem hiding this comment.
Fixed the all-or-none placement. insertFilterBeforeAllWindows now refuses a stop point while any Window remains below the candidate filter. Added generator regressions where HAVING references the lower window alias with both SELECT-list orders, under both config values. In every case the filter remains above the full window chain and group_count is 2.
| /** | ||
| * Extracts all nested window expressions and places the HAVING filter below every Window. | ||
| */ | ||
| private def addWindowWithHaving( |
There was a problem hiding this comment.
Non-blocking (P2): This placement runs only when HAVING reaches ExtractWindowExpressions still attached to its Aggregate, which grouping analytics does only for conditions that need type coercion. BaseGroupingSets.resolved is always false, so the UnresolvedHaving case that calls this method cannot match while the Aggregate still has ROLLUP, CUBE or GROUPING SETS. ResolveGroupingAnalytics, which runs earlier in the batch, calls the unchanged tryResolveHavingCondition instead. When the condition resolves without type coercion, that method returns Filter(cond, constructAggregate(...)) while the window expressions are still in the aggregate list. Every window layer is then extracted below that Filter, whatever spark.sql.legacy.evaluateHavingAfterInnerWindow is set to. For example:
SELECT k, SUM(RANK() OVER (ORDER BY SUM(v) DESC)) OVER () AS rank_sum
FROM VALUES ('drop', 2), ('keep', 1) AS input(k, v)
GROUP BY ROLLUP(k)
HAVING k = 'keep'The windows run over the groups (NULL, 3), ('drop', 2) and ('keep', 1), so the query returns (keep, 6). Evaluating HAVING first gives (keep, 1), which is what the same query with GROUP BY k now returns. The existing GROUPING_ID() = 1 golden lands below its window only because that condition needs coercion and is deferred to this method. The migration note and the config doc state the fix unconditionally, and no test covers this form. A HAVING condition that reads a SELECT-list window alias also resolves on this route and is evaluated after every window; that case is outside this gap.
Recommended change: In ResolveGroupingAnalytics, when spark.sql.legacy.evaluateHavingAfterInnerWindow is false, the aggregate list still contains window expressions, and the resolved HAVING condition reads no window output, keep the resolved HAVING attached to the constructed Aggregate as an unresolved HAVING, instead of turning it into a Filter. The nested-window placement then extracts every window layer first and puts the Filter directly on the Aggregate. A condition that reads a SELECT-list window alias keeps the current Filter above every window. Add window-resolution SQL golden tests for ROLLUP or GROUPING SETS with nested and single-level windows and a HAVING that needs no coercion, plus one whose HAVING reads a window alias, under both flag values, and update the migration note and the config doc to cover grouping analytics.
Why this works: The condition is already resolved and remapped to the constructed Aggregate's output, and that Aggregate's own expressions are resolved. ExtractWindowExpressions' HAVING case therefore matches it in the same analyzer batch, and the nested-window placement builds the window layers over a Filter on the Aggregate. The existing grouping-function and aggregate rewrites still see a Filter directly over an Aggregate. When extra aggregates were pushed for the condition, the existing projection that drops them stays on top. Because this rule resolves HAVING against the Aggregate output, which includes the SELECT-list window aliases, the remapped condition's references are checked against the outputs of the window-bearing aggregate expressions: a condition that reads one keeps the current Filter, since the nested-window placement would put it below the Window that produces that value and analysis would fail with a missing attribute.
Scope: Route grouping-analytics HAVING that reads no window output through the change's nested-window placement, add goldens for that route, and align the documentation.
Compatibility: Grouping-analytics HAVING without window functions in the SELECT list keeps its current plans. A grouping-analytics HAVING that reads a window alias keeps its Filter above every window, and grouping-analytics queries with a generator in the SELECT list never reach this rule, so both keep their current plans.
Risks: Single-level window queries over ROLLUP, CUBE or GROUPING SETS with a HAVING that needs no coercion also change plans and results. HAVING now runs before their window, which corrects the base-branch behavior. The legacy flag rolls this back, and the migration note should mention it.
Constraints: Gate the new routing on spark.sql.legacy.evaluateHavingAfterInnerWindow, so that true keeps the current placement on this route. Keep the HAVING condition resolved against the constructed Aggregate, including the existing attribute remapping and any aggregate expressions pushed for the condition. Do not defer a condition that reads a window output; keep it as a Filter above every window, as today.
Exclusions:
- The single-pass resolver already declines HAVING with grouping analytics and falls back to the fixed-point analyzer, and it stays unchanged.
- Whether a HAVING condition that reads a window alias should be rejected on this route, as it already is for plain GROUP BY, is a separate decision that this repair does not make.
Success: With the legacy flag false, ROLLUP, CUBE and GROUPING SETS queries with window functions and a HAVING condition that resolves without coercion and reads no window output evaluate HAVING below every window layer. With the legacy flag true, those queries keep the base-branch plans and results. A grouping-analytics HAVING condition that reads a SELECT-list window alias keeps its current plan and result under both flag values. The migration note and the config doc accurately describe grouping-analytics queries.
Verification:
- Regression: A ROLLUP or GROUPING SETS query with nested window functions and a HAVING on a grouping key that needs no coercion places the HAVING filter directly above the Aggregate and returns rank_sum 1 by default.
- Compatibility: The same query with spark.sql.legacy.evaluateHavingAfterInnerWindow set to true keeps the base-branch plan and result.
- Behavior: Grouping-analytics HAVING conditions that use grouping functions, aggregates absent from the SELECT list, or a subquery still resolve and keep the right groups when window functions are present.
- Behavior: A single-level window over ROLLUP, CUBE or GROUPING SETS with a HAVING that needs no coercion evaluates HAVING before the window by default.
- Behavior: A ROLLUP or GROUPING SETS query with window functions whose HAVING reads a SELECT-list window alias without needing coercion keeps the HAVING filter above every window and returns the same result as before the change, under both flag values.
There was a problem hiding this comment.
Fixed the grouping-analytics path. With spark.sql.legacy.evaluateHavingAfterInnerWindow=false, tryResolveHavingCondition keeps a resolved UnresolvedHaving only when the aggregate contains window outputs and the condition does not reference one of them; ExtractWindowExpressions then places the filter directly above the constructed aggregate. Conditions referencing a window output keep the previous analysis behavior. Added ROLLUP nested-window, GROUPING SETS single-window, window-alias negative, grouping-function, HAVING-only aggregate, and scalar-subquery coverage, plus legacy-true cases. Default results are rank_sum = 1 and group_count = 1; legacy results remain 6 and 3. The config doc and migration guide now cover grouping analytics.
There was a problem hiding this comment.
Thanks, the deferral fixes the main cases: by default these conditions now sit directly above the constructed aggregate, below every window layer, and the legacy goldens keep the old plans. This review raises two related gaps: a HAVING condition with a nondeterministic expression still runs after the windows, and the check for window outputs also counts window functions inside SELECT-list subqueries.
| } // end of addWindow | ||
|
|
||
| /** | ||
| * Extracts all nested window expressions and places the HAVING filter below every Window. |
There was a problem hiding this comment.
Nit (P3): With LEGACY_EVALUATE_HAVING_AFTER_INNER_WINDOW set to true, the recursion guard below fails. The method then extracts only the outermost layer and returns addWindow over Filter(condition, Aggregate(...)), whose aggregate list still holds the inner window expressions. Those are later extracted below the filter, as the new legacy golden shows (Window [sum] +- Filter (k = keep) +- Project +- Project +- Window [rank] +- Aggregate). The doc states the default-branch guarantee unconditionally; it could cover both branches:
| * Extracts all nested window expressions and places the HAVING filter below every Window. | |
| * Extracts all nested window expressions and places the HAVING filter below every Window, | |
| * unless LEGACY_EVALUATE_HAVING_AFTER_INNER_WINDOW is true: then only the outermost layer is | |
| * extracted here, and the inner windows are later extracted below the HAVING filter. |
Verification:
- Inspection: The addWindowWithHaving Scaladoc states that HAVING goes below every nested window layer only when spark.sql.legacy.evaluateHavingAfterInnerWindow is false, and that with true only the outermost layer is extracted and HAVING can end up above the inner windows.
There was a problem hiding this comment.
Updated the Scaladoc to describe both branches: the default extracts every nested window layer before placing HAVING, while the legacy setting extracts only the outermost layer here and allows inner windows to be extracted below the filter later.
spark-test
left a comment
There was a problem hiding this comment.
Review summary
The earlier findings are resolved at this head. Generator-path HAVING no longer lands between two windows, grouping-analytics HAVING that needs no cast now goes below every window layer, and the addWindowWithHaving Scaladoc covers both branches. The plain GROUP BY nested-window placement works as described, and the legacy goldens reproduce the base-branch plans and results.
The new all-or-none check in the generator path looks beyond the window chain it guards. By default, any window inside the aggregate's own input keeps HAVING above every SELECT-list window, so those windows silently count removed groups. Examples are a window in a derived table, view, join input, or WHERE subquery. Single-window generator queries of that shape returned correct results on the base branch.
The new grouping-analytics deferral makes ROLLUP, CUBE, and GROUPING SETS queries fail analysis when they combine a lateral column alias, a window, and a HAVING condition that needs no cast. These queries analyzed on the base branch. The goldens added for that branch use conditions that need casts, so they take the older route, and the branch's window-output, HAVING-only-aggregate, and lateral-alias paths stay untested.
The insertFilterBeforeWindow Scaladoc also does not describe the legacy-flag dispatch.
Cross-cutting observations
-
Both regressions came from the fixes for the previous review. HAVING placement is now decided per route, and each route relies on a plan shape that other rules also own: the generator-path placement in ResolveUnresolvedHaving inspects the aggregate's whole input, and the grouping-analytics branch relies on how window extraction handles an aggregate with lateral aliases.
The lateral-alias fix sends its queries through ResolveUnresolvedHaving's placement, so those queries over inputs that contain windows get HAVING before their windows only once the generator-path fix also lands.
Findings
4 total: 0 P0, 1 P1, 1 P2, 2 P3.
Blocking (P1)
- Generator HAVING stays above the windows when the aggregate's input has a window —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4986— see inline.
Non-blocking (P2)
- Grouping-analytics HAVING with a lateral alias and a window now fails analysis —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:988— see inline.
Nit (P3)
- Grouping-analytics goldens do not reach the new HAVING branch —
sql/core/src/test/resources/sql-tests/inputs/window-resolution-edge-cases.sql:177— see inline. - insertFilterBeforeWindow Scaladoc omits the legacy-flag dispatch —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4966— see inline.
Shared repair plans
Shared repair plan 1
Covered findings:
- Grouping-analytics HAVING with a lateral alias and a window now fails analysis —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:988 - Grouping-analytics goldens do not reach the new HAVING branch —
sql/core/src/test/resources/sql-tests/inputs/window-resolution-edge-cases.sql:177
Recommended change: In tryResolveHavingCondition, the new branch defers a grouping-analytics HAVING by emitting UnresolvedHaving over the constructed aggregate. When that aggregate's list still holds lateral column alias references, first apply the lateral-alias lift to it, the same rewrite ResolveLateralColumnAliasReference performs for an aggregate under a Filter, and then wrap the resulting Project in UnresolvedHaving. Keep emitting the Filter, as the base branch does, when the aggregate is not eligible for that lift or the condition uses GROUPING() or GROUPING_ID(), and leave a TODO at that exemption, because no rule after the main resolution batch rewrites grouping functions. Also add window-resolution golden cases (analyzer plans and results), by default and with the legacy flag, that reach the branch: a ROLLUP or GROUPING SETS query with a lateral alias, a window, and a type-matched HAVING; a HAVING that reads a window output with a matching literal type (for example group_count = 3L); a HAVING-only aggregate (SUM(v) = 1L); and a grouping function (GROUPING_ID() = 1L).
Why this works: Lifting resolves the lateral aliases before HAVING is deferred, so no consumer sees them. Because the HAVING child is now a Project, the HAVING case of window extraction no longer matches, and the windows are extracted through its Project case instead. After the main resolution batch, ResolveUnresolvedHaving inserts HAVING at the lifted Project or at the aggregate below the windows, depending on which node outputs the condition's references. Aggregates without lateral aliases keep the current direct route, where window extraction turns HAVING into a Filter that the grouping-function rewrite still sees.
Scope: Resolve lateral aliases before deferring grouping-analytics HAVING, keep the base-branch Filter where the deferred route cannot finish the condition, and cover every path of the deferral branch with goldens.
Compatibility: Lateral-alias handling for plain GROUP BY, the grouping-analytics route for conditions that need casts, and lateral-alias queries whose HAVING uses a grouping function or whose aggregate the lift does not accept all keep their base-branch behavior.
Risks: Grouping-analytics queries with lateral aliases, windows, and a type-matched HAVING move from the base-branch plan, where HAVING ran after the windows, to HAVING before the windows, so their window values can change. The legacy flag restores the base-branch plan. Lateral-alias queries whose type-matched HAVING uses GROUPING() or GROUPING_ID() keep the base-branch placement, so their windows can still see groups that HAVING removes until the TODO is resolved.
Constraints: Keep the condition remapped to the constructed aggregate's output, including any aggregates the condition pushed, and make sure the lift preserves those output attributes. Do not change how ResolveLateralColumnAliasReference handles an aggregate under UnresolvedHaving for plain GROUP BY. With the legacy flag set to true, keep emitting a Filter, as the base branch does. Send only a condition without GROUPING() or GROUPING_ID() through ResolveUnresolvedHaving, since no later rule rewrites grouping functions and CheckAnalysis rejects any that remain.
Exclusions:
- Conditions that need a cast keep taking the older deferred route. Their lateral-alias rejection already exists on the base branch and is unchanged.
- Plain GROUP BY queries that combine lateral aliases, windows, and HAVING keep their documented UNSUPPORTED_FEATURE error.
- A grouping-analytics query whose lateral-alias aggregate the lift does not accept, or whose type-matched HAVING uses GROUPING() or GROUPING_ID(), keeps the base-branch Filter and its analysis outcome. The TODO at that exemption names the missing grouping-function rewrite after the main resolution batch.
Success: By default, a ROLLUP, CUBE, or GROUPING SETS query that combines a lateral column alias, a window function, and a type-matched HAVING without grouping functions analyzes and evaluates HAVING before every window. Goldens reach the deferral branch's window-output Filter, its HAVING-only-aggregate projection, and its grouping-function handling, and pin their default and legacy results. A grouping-analytics query with a lateral alias whose type-matched HAVING uses a grouping function keeps its base-branch plan and analysis outcome. With the legacy flag set to true, grouping-analytics plans and results match the base branch.
Verification:
- Regression: SELECT k, SUM(v) AS s, s * 2 AS s2, SUM(SUM(v)) OVER (PARTITION BY k) AS ps FROM VALUES ('a', 1), ('b', 2) AS t(k, v) GROUP BY ROLLUP(k) HAVING k IS NOT NULL analyzes by default and returns the rows for 'a' and 'b', with ps equal to each group's own sum.
- Regression: By default, a grouping-analytics query with a lateral alias, COUNT(*) OVER (), and a HAVING that removes a group counts only the surviving groups.
- Behavior: A type-matched grouping-analytics HAVING that reads a window output keeps HAVING above the windows and returns the base-branch result. A type-matched HAVING-only aggregate and a type-matched grouping function are evaluated below the window, and the pushed aggregate is projected away.
- Compatibility: With the legacy flag set to true, each of these grouping-analytics queries keeps its base-branch plan and result, or its base-branch analysis outcome.
- Compatibility: Plain GROUP BY queries that combine a lateral alias, a window, and HAVING still fail with UNSUPPORTED_FEATURE.LATERAL_COLUMN_ALIAS_IN_AGGREGATE_WITH_WINDOW_AND_HAVING.
- Compatibility: By default, a ROLLUP or GROUPING SETS query with a lateral alias, a window, and a type-matched HAVING that uses GROUPING_ID() (for example GROUPING_ID() = 1L) keeps its base-branch plan and analysis outcome.
Re-review status
Prior AI findings: 3 no longer raised, 0 still raised; additional unresolved findings in this review: 4.
New attribution: 3 newly introduced, 1 late catch, 0 previously raised, 0 unattributed.
Generated by Omnigent on Databricks
| case window: Window if filter.condition.references.subsetOf(window.child.outputSet) => | ||
| insertFilterBeforeAllWindows(filter.copy(child = window.child), foundWindow = true) | ||
| .map(child => window.withNewChildren(Seq(child))) | ||
| case _ if foundWindow && !filter.child.containsPattern(WINDOW) && |
There was a problem hiding this comment.
Blocking (P1): In default mode this guard checks the stop node's whole subtree, not just the SELECT-list window chain. Tree-pattern bits are unioned over children, expressions, and subquery plans, so any Window below the grouping Aggregate makes the walk return None. That includes a FROM-clause derived table or view with a window function, a windowed join input, and a WHERE subquery that uses one. ResolveUnresolvedHaving then keeps HAVING above every SELECT-list window, and those windows count the groups that HAVING removes. For example:
SELECT explode(array(k)) AS g, COUNT(*) OVER () AS c, k
FROM (SELECT k, v, ROW_NUMBER() OVER (ORDER BY v) AS rn
FROM VALUES ('drop', 2), ('keep', 1) AS t(k, v)) s
GROUP BY k
HAVING k = 'keep'This returns (keep, 2, keep) by default. The base branch, and this PR with spark.sql.legacy.evaluateHavingAfterInnerWindow=true, return (keep, 1, keep), as the Spark 4.3 migration note promises for generator queries. With nested SELECT-list windows over such an input, every layer now sees the removed groups. No generator golden reads from a windowed input.
Suggested fix: bound the check to the chain the walk descends through. Accept the stop node unless a Window is reachable from it through Project, Generate, and Window nodes. That still rejects a stop at or above a lower window of the chain, and it no longer looks inside the aggregate's input.
This extends an existing discussion.
Verification:
- Regression: A generator query with COUNT(*) OVER () and a HAVING on the grouping key evaluates HAVING before the window whenever the aggregate's input contains a window function: when it reads from a derived table that uses ROW_NUMBER(), and when its WHERE clause has a subquery that uses a window function. For the ('drop', 2), ('keep', 1) input, with both rows kept by the WHERE clause, the window counts 1, not 2.
- Regression: The same windowed-input shape with nested SELECT-list windows places HAVING below every window layer and returns the corrected nested result.
- Behavior: Generator queries whose HAVING reads a lower window's alias keep HAVING above the whole window chain in both SELECT-list orders.
- Compatibility: With the legacy flag set to true, the windowed-input generator queries keep their base-branch plans and results.
There was a problem hiding this comment.
Addressed at the current head. The stop check now follows only the SELECT-list Project/Generate/Window spine, so a window in a derived-table input or WHERE subquery no longer blocks HAVING placement. I added default and legacy goldens for windowed inputs, windowed subqueries, and nested SELECT-list windows over a windowed input.
| val withHaving = if ( | ||
| !conf.getConf(SQLConf.LEGACY_EVALUATE_HAVING_AFTER_INNER_WINDOW) && | ||
| windowOutputSet.nonEmpty && newCond.references.intersect(windowOutputSet).isEmpty) { | ||
| UnresolvedHaving(newCond, newChild) |
There was a problem hiding this comment.
Non-blocking (P2): This new UnresolvedHaving breaks grouping-analytics queries that use a lateral column alias. ResolveLateralColumnAliasReference deliberately leaves an Aggregate under UnresolvedHaving untouched, and a LateralColumnAliasReference counts as resolved. Later in the same batch, the HAVING case of ExtractWindowExpressions matches this aggregate and throws UNSUPPORTED_FEATURE.LATERAL_COLUMN_ALIAS_IN_AGGREGATE_WITH_WINDOW_AND_HAVING. On the base branch, the Filter parent let the lateral-alias rule lift the aggregate into a Project, and the query analyzed. For example, this query returns the a and b rows on the base branch and now fails by default:
SELECT k, SUM(v) AS s, s * 2 AS s2, SUM(SUM(v)) OVER (PARTITION BY k) AS ps
FROM VALUES ('a', 1), ('b', 2) AS t(k, v)
GROUP BY ROLLUP(k)
HAVING k IS NOT NULLIts base-branch rows were already correct, because dropping the grand-total group cannot change another partition. The migration note does not mention the new error, and no golden uses a lateral alias on this route.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
Addressed at the current head. Grouping-analytics deferral now resolves lateral column aliases to a fixed point before wrapping the plan in DeferredHaving. If the lift cannot complete, or the condition uses a grouping function, it retains the existing Filter path. Default and legacy lateral-alias goldens cover the corrected behavior.
| COUNT(*) OVER () AS group_count | ||
| FROM VALUES ('drop', 2), ('keep', 1) AS input(k, v) | ||
| GROUP BY GROUPING SETS ((k), ()) | ||
| HAVING group_count = 3 |
There was a problem hiding this comment.
Nit (P3): group_count = 3 compares a bigint with an int literal, so it never reaches the new branch in tryResolveHavingCondition. Type coercion skips an operator whose child is the still-unresolved GROUPING SETS aggregate. The condition is therefore unresolved when tryResolveHavingCondition checks it, the method returns early, and the condition takes the older deferred route. That is why this case fails with the same MISSING_ATTRIBUTES error under both flag values. GROUPING_ID() = 1 and SUM(v) = 1 take the same route. The only grouping-analytics cases that reach the branch are k = 'keep' and k = (SELECT 'keep'). No golden would fail if the branch's Filter for a window-output reference, its Project that drops HAVING-only aggregates, or its lateral-alias handling regressed.
This extends existing discussions 1, 2.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
Addressed. The grouping-analytics cases now use type-matched 3L/1L literals so they reach the new branch in tryResolveHavingCondition. The goldens cover the window-output path, HAVING-only aggregate projection, grouping-function handling, and lateral aliases under both default and legacy settings.
There was a problem hiding this comment.
Thanks, the type-matched literals helped: the window-output case now analyzes, the grouping-function filter stays above the window, and the lateral-alias plans change with the flag, so those cases reach the new branch. The HAVING-only aggregate case still does not: HAVING SUM(v) = 1L has the same plan by default and with the legacy flag, so no golden reaches the projection that drops HAVING-only aggregates above the deferred HAVING. This review raises that remaining gap, with its cause and a case that does reach the branch.
| private def insertFilterBeforeWindow(filter: Filter): Option[LogicalPlan] = filter.child match { | ||
| case project: Project => | ||
| insertFilterBeforeWindow(filter.copy(child = project.child)) | ||
| private def insertFilterBeforeWindow(filter: Filter): Option[LogicalPlan] = { |
There was a problem hiding this comment.
Nit (P3): The Scaladoc above this method is unchanged from the base branch. It still describes a single search through Project and Generate nodes for "a Window chain". The method now dispatches on spark.sql.legacy.evaluateHavingAfterInnerWindow. By default, insertFilterBeforeAllWindows also descends through Window nodes and the Projects between window layers, and either places HAVING below every SELECT-list window or leaves it above all of them. With the flag set to true, insertFilterBeforeOuterWindow places it only below the first contiguous run of Window operators, which can put it between two layers. For a nested generator chain, a reader cannot tell which of those two plans the method produces. The Scaladoc could describe both branches, as the addWindowWithHaving Scaladoc now does.
Verification:
- Inspection: The insertFilterBeforeWindow Scaladoc names the legacy flag, describes the placement each value produces, and agrees with the stop check being limited to the window chain.
There was a problem hiding this comment.
Addressed. The Scaladoc now documents both dispatch paths: the default path places HAVING below the complete SELECT-list window chain or leaves it above all of it, while the legacy path moves it below only the first contiguous Window run.
spark-test
left a comment
There was a problem hiding this comment.
Review summary
The earlier findings are resolved at this head, apart from one test gap. A window in the aggregate's own input no longer keeps generator-query HAVING above the SELECT-list windows, grouping-analytics queries that combine a lateral column alias, a window and a HAVING without a cast analyze again because the aliases are lifted before deferral, and the insertFilterBeforeWindow Scaladoc now describes the legacy dispatch. Of the grouping-analytics goldens, the window-output, grouping-function and lateral-alias cases now reach the new branch, but the HAVING-only aggregate case still takes the older route.
On the ROLLUP, CUBE and GROUPING SETS route, the new deferral breaks queries that analyzed on the base branch. By the review's reading of the analyzer, such a query with a SELECT-list window, a HAVING that needs no cast and a QUALIFY without its own window function now fails analysis with QUALIFY_REQUIRES_WINDOW_FUNCTION by default, including queries whose results were already correct on the base branch. A v2 CREATE TABLE AS SELECT or REPLACE TABLE AS SELECT of such a query fails with an internal planner error, and view and cache DDL whose HAVING holds an IN, EXISTS or correlated scalar subquery fails at creation. A HAVING condition with a nondeterministic expression still runs after the windows, although the migration note says such conditions now run first.
The check for window outputs also counts window functions inside SELECT-list subqueries. A grouping-analytics query whose only window is inside such a subquery now fails analysis when it orders by an aggregate it does not select, and a HAVING on such a subquery alias still runs after the real windows.
The remaining findings are test and layout gaps: no golden reaches the projection that drops HAVING-only aggregates or the lateral-alias paths for chained aliases and a HAVING on a lifted alias, and the new tests split an existing QUALIFY section.
Review-level concerns
- Shared problem: The grouping-analytics fix defers HAVING in a resolved DeferredHaving node that stays in the plan until the last analyzer batch, so every rule that runs in between and expects either the HAVING Filter or a finished plan acts on it without recognizing it: QUALIFY resolution in the same batch, the nondeterministic pull-out in a later batch, and the analysis check and freeze of CREATE TABLE AS SELECT, view and cache commands. Fixing those rules one by one would not be enough, because ORDER BY resolution, outer-reference updates and rules injected by extensions face the same node, and the previous two revisions of this route each moved the failure to another rule that relies on the HAVING's plan shape. The boundary-level response is to give the marker one short lifetime, ending in the main resolution batch as soon as window extraction finishes, so that the remaining batches and commands see only the placed Filter. The QUALIFY and command failures are blocking.
Cross-cutting observations
-
Placing the deferred HAVING right after window extraction would also end the ORDER BY failure for queries whose only window sits inside a SELECT-list subquery, because the Sort would then see a Filter over the aggregate, but it would not move a HAVING that reads such a subquery alias below the real windows, since that depends on which items count as window outputs rather than on how long the marker lives.
The window-output fix is still needed after the marker fix, and it can be made first or on its own.
Findings
7 total: 0 P0, 3 P1, 1 P2, 3 P3.
Blocking (P1)
- CTAS, view and cache commands freeze the deferred HAVING before it is placed —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:5043— see inline. - QUALIFY without its own window fails over a deferred grouping-analytics HAVING —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1065— see inline. - Window-output test counts windows inside SELECT-list subqueries —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1047— see inline.
Non-blocking (P2)
- Nondeterministic grouping-analytics HAVING still runs after the windows —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:5022— see inline.
Nit (P3)
- HAVING-only aggregate golden still misses the deferred branch (still raised) —
sql/core/src/test/resources/sql-tests/inputs/window-resolution-edge-cases.sql:217— see inline. - Lateral-alias goldens miss chained aliases and HAVING on a lifted alias —
sql/core/src/test/resources/sql-tests/inputs/window-resolution-edge-cases.sql:171— see inline. - New HAVING tests split the QUALIFY-with-HAVING section —
sql/core/src/test/resources/sql-tests/inputs/window-resolution-edge-cases.sql:103— see inline.
Shared repair plans
Shared repair plan 1
Covered findings:
- CTAS, view and cache commands freeze the deferred HAVING before it is placed —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:5043 - Nondeterministic grouping-analytics HAVING still runs after the windows —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:5022 - QUALIFY without its own window fails over a deferred grouping-analytics HAVING —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1065
Recommended change: Convert the deferred grouping-analytics HAVING into its placed Filter inside the main resolution batch, as soon as window extraction has produced the marker's child, by running the existing all-or-none placement walk there, for example in a rule that runs right after ExtractWindowExpressions in the same batch, and make DeferredHaving report itself unresolved until that conversion so that no rule acts on it in between; the last-batch DeferredHaving case then only needs to cover markers the main batch could not convert, or can be removed. Keep the marker and its lateral-alias lift rather than returning to the UnresolvedHaving deferral of commit 02fb393: the ExtractWindowExpressions case that placed that HAVING inside the main batch matches only a HAVING directly over an Aggregate, so once the lift has put a Project above the aggregate the HAVING would again wait for the last batch, where a QUALIFY above it never resolves and an analysis-only command over it is never marked analyzed. Add window-resolution SQL golden tests for QUALIFY over such a query, with and without a lateral column alias, for a seeded nondeterministic HAVING over grouping analytics, and for view and cache DDL whose HAVING holds an IN subquery, and DataSource V2 SQL tests for CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT of such a query, with and without a lateral column alias.
Why this works: The marker only needs to carry the resolved condition while window extraction builds the chain under it. Placing the Filter in the same batch right after extraction, with the walk the last batch uses today, gives every later rule the ordinary HAVING Filter below the window chain: ResolveQualify finds the windows through it, PullOutNondeterministic skips it, HandleSpecialCommand checks and freezes a finished plan, and ORDER BY resolution and UpdateOuterReferences see a Filter over an Aggregate when no window was extracted. Keeping the marker unresolved until then makes its parents wait instead of acting on it, the way they wait on UnresolvedHaving.
Scope: Move the deferred grouping-analytics HAVING's placement into the main resolution batch and keep the marker unresolved until it is placed.
Compatibility: Apart from what the separate fix for windowed SELECT-list subqueries changes, every query the current change already places keeps the same placement and result: only when the placement runs changes, not the walk, the lateral-alias lift or the grouping-function exemption.
Risks: Parents of a deferred grouping-analytics HAVING now wait for the placed Filter, which can add one Resolution batch iteration for these queries; the legacy flag restores the base path. Rules in the remaining Resolution batch iterations now see the HAVING Filter directly over the aggregate below the window chain, the shape plain GROUP BY already produces, with a condition that is already resolved and remapped.
Constraints: Keep the all-or-none placement walk, the lateral-alias lift and the grouping-function exemption as they are, and leave which SELECT-list items count as window outputs to the separate fix for windowed SELECT-list subqueries, so this change alters only when the placement runs. With spark.sql.legacy.evaluateHavingAfterInnerWindow set to true, create no marker and keep the base plans. Leave the generator route's UnresolvedHaving and its last-batch placement unchanged.
Exclusions:
- Which SELECT-list items count as window outputs is a separate repair: the check that also counts window functions inside SELECT-list subqueries.
- Grouping-function conditions stay on the base Filter path, as the TODO at that exemption defers.
Success: By default, a ROLLUP, CUBE or GROUPING SETS query with a SELECT-list window, a HAVING that resolves without a cast and a QUALIFY with no window function of its own, with or without lateral column aliases, analyzes and evaluates HAVING before the windows; the ROLLUP query with COUNT(*) OVER () AS group_count, HAVING k = 'keep' and QUALIFY group_count > 0 returns (keep, 1). A HAVING condition with a nondeterministic conjunct over grouping analytics is evaluated below the SELECT-list windows, as the same condition without it is. CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT of such a query, with or without lateral column aliases, into a v2 catalog write the corrected rows, and CREATE VIEW, ALTER VIEW ... AS and CACHE TABLE ... AS SELECT accept such a HAVING with an IN or EXISTS subquery. After the main resolution batch, no deferred grouping-analytics HAVING remains in the plan or in the query of an analysis-only command, whether as DeferredHaving or as UnresolvedHaving, including when its lateral column aliases were lifted. With the legacy flag set to true, plans and results stay those of the base branch.
Verification:
- Regression: By default, a ROLLUP or GROUPING SETS query with a SELECT-list window, a cast-free HAVING on the grouping key and a QUALIFY that reads only a SELECT-list window alias analyzes and returns the HAVING-filtered rows, the ROLLUP query whose QUALIFY is group_count > 0 returns (keep, 1), and a QUALIFY that reads a grouping column outside the SELECT list resolves; the same holds when the SELECT list also has a lateral column alias, such as SUM(v) AS s, s * 2 AS s2.
- Regression: A ROLLUP query with COUNT(*) OVER () and HAVING k = 'keep' plus an always-true seeded rand() conjunct returns group_count 1 by default, the same as without the conjunct, and its analyzed plan places the filter below the window.
- Regression: CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT into a v2 catalog of a ROLLUP query with COUNT(*) OVER () and HAVING k = 'keep' succeed and write (keep, 1), and CREATE VIEW, ALTER VIEW ... AS and CACHE TABLE ... AS SELECT of such a query with HAVING k IN (SELECT 'keep') succeed and read back (keep, 1); CREATE TABLE AS SELECT of the ROLLUP query that also selects SUM(v) AS s and s * 2 AS s2 succeeds and writes the corrected row.
- Compatibility: With the legacy flag set to true, each of these statements keeps its base-branch plan, result or analysis outcome, and the existing window-resolution and generator goldens keep their current default and legacy plans and results.
- Inspection: After the main resolution batch no deferred grouping-analytics HAVING, as DeferredHaving or as UnresolvedHaving, remains in the plan, including inside analysis-only commands and for queries whose lateral column aliases were lifted, and no rule acts on the marker while it is present.
Re-review status
Prior AI findings: 3 no longer raised, 1 still raised; additional unresolved findings in this review: 6.
New attribution: 4 newly introduced, 2 late catch, 0 previously raised, 0 unattributed.
Generated by Omnigent on Databricks
| if havingCondition.resolved && child.resolved => | ||
| val filter = Filter(condition = havingCondition, child = child) | ||
| insertFilterBeforeWindow(filter).getOrElse(filter) | ||
| case d @ DeferredHaving(condition, child) if d.resolved => |
There was a problem hiding this comment.
Blocking (P1): This case runs in the last analyzer batch, after HandleSpecialCommand. That rule calls checkAnalysis on every resolved analysis-only command and marks its query analyzed, which turns the query into an inner child. ResolveUnresolvedHaving reaches neither inner children nor analyzed subtrees, so a DeferredHaving inside a command's query is never converted, and no optimizer rule or planner strategy handles it.
By the review's reading of the code, with default settings and a v2 catalog testcat:
CREATE TABLE testcat.ns.t AS
SELECT k, COUNT(*) OVER () AS c
FROM VALUES ('drop', 2), ('keep', 1) AS t(k, v)
GROUP BY ROLLUP(k)
HAVING k = 'keep'CreateTableAsSelectExec creates the table, then writeToTable runs the frozen query through a new QueryExecution whose analysis skips it again, so planning fails with No plan for DeferredHaving, reported as an internal error, and the table is dropped. REPLACE TABLE AS SELECT takes the same path. On the base branch the HAVING is already a Filter when the command is checked, and the statement writes its row.
The same early check also validates the marker as a subquery host: CREATE VIEW, ALTER VIEW ... AS and CACHE TABLE ... AS SELECT of such a query whose HAVING has an IN, EXISTS or correlated scalar subquery fail at creation with UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY, although the bare SELECT runs. With spark.sql.legacy.storeAnalyzedPlanForView=true, temporary views and CACHE TABLE ... AS SELECT store the unconverted plan, which fails when read. v1 data source and Hive CTAS are not affected, because they re-analyze their query when they write. No test runs a command over a deferred HAVING.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
Fixed in b970dec. DeferredHaving now remains unresolved and is placed immediately after ExtractWindowExpressions in the main resolution batch, before HandleSpecialCommand can freeze the query. Added v2 CTAS/RTAS coverage, including a lateral-alias CTAS, plus CREATE/ALTER VIEW and CACHE TABLE cases with HAVING subqueries.
| val remappedCondition = newCond.transform { | ||
| case a: AttributeReference => liftedAttrMap.getOrElse(a, a) | ||
| } | ||
| DeferredHaving(remappedCondition, liftedChild) |
There was a problem hiding this comment.
Blocking (P1): The parser puts QUALIFY directly over HAVING, and ResolveQualify runs in the main batch right after window extraction, while the DeferredHaving created here stays until the last batch. Once window extraction resolves the marker's child, ResolveQualify fires: resolveQualifyCondition treats the marker as a leaf, and hasWindowInPlan recognizes only Window, Project and Filter, so it does not see the windows below. A QUALIFY condition without its own window function then throws QUALIFY_REQUIRES_WINDOW_FUNCTION, whose message wrongly says the query has no window function.
By the review's reading of the code, with default settings this query fails analysis:
SELECT k, RANK() OVER (PARTITION BY k ORDER BY SUM(v)) AS rnk
FROM VALUES ('a', 1), ('b', 2) AS input(k, v)
GROUP BY ROLLUP(k)
HAVING k IS NOT NULL
QUALIFY rnk = 1On the base branch the child is a Filter over the window chain, so it would return (a, 1) and (b, 1), which is also the correct result because each partition holds one group. Queries this PR sets out to fix fail the same way, such as one with COUNT(*) OVER () AS group_count, HAVING k = 'keep' and QUALIFY group_count > 0. Because the leaf case also stops at the marker, a QUALIFY that reads a grouping column outside the SELECT list, which the base branch adds to the aggregate, now stays unresolved. This holds with or without lateral column aliases, and no golden combines grouping analytics, a deferrable HAVING, a window and QUALIFY.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
Fixed in b970dec. ResolveDeferredHaving now runs immediately after ExtractWindowExpressions and before ResolveQualify, so QUALIFY sees the placed Filter and window chain. Added regressions both without lateral aliases and with lateral aliases plus a grouping column outside the SELECT list.
| */ | ||
| private[analysis] case class DeferredHaving( | ||
| condition: Expression, | ||
| child: LogicalPlan) extends OrderPreservingUnaryNode { |
There was a problem hiding this comment.
Non-blocking (P2): Because this marker is a resolved unary node whose output is its child's, PullOutNondeterministic, which runs before the last batch, rewrites it as it rewrites any such node: it moves the nondeterministic expression into a new Project below the marker and makes the condition read the new _nondeterministic attribute. insertFilterBeforeAllWindows then cannot descend past that Project, because the attribute is not in its child's output, and keeps the Filter above every window. By the review's reading of the code, this query would still return group_count = 3, as on the base branch:
SELECT k, COUNT(*) OVER () AS group_count
FROM VALUES ('drop', 2), ('keep', 1) AS input(k, v)
GROUP BY ROLLUP(k)
HAVING k = 'keep' AND rand() < 2DWithout rand(), or with GROUP BY k, it would return 1. The migration note and the config doc say such conditions now run before the windows, and the PR defers only grouping functions. Plain GROUP BY and generator queries are not affected, because PullOutNondeterministic leaves a Filter and an unresolved node alone. No test covers a nondeterministic HAVING on this route.
This extends an existing discussion.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
Fixed in b970dec. The deferred marker is converted to a Filter in the main resolution batch before PullOutNondeterministic runs, so the seeded RAND predicate remains below the SELECT-list window chain. Added default and legacy golden coverage.
|
|
||
| val windowOutputSet = AttributeSet( | ||
| aggregate.aggregateExpressions.zip(newChild.output).collect { | ||
| case (expr, output) if expr.containsPattern(WINDOW_EXPRESSION) => output |
There was a problem hiding this comment.
Blocking (P1): containsPattern(WINDOW_EXPRESSION) reads tree-pattern bits, and a subquery expression's bits include its plan's, so a SELECT-list subquery that uses a window function counts as a window output here. Window extraction and Aggregate.resolved walk only the expression tree and treat that item as an ordinary expression. By the review's reading of the code, two effects follow.
When such a subquery is the only window-bearing item, HAVING is deferred over an aggregate that no window extraction touches, so DeferredHaving sits directly on the resolved Aggregate until the last batch. ResolveReferencesInSort and ResolveAggregateFunctions resolve ORDER BY aggregates only under an Aggregate or a Filter over one, so this query, which analyzed on the base branch, now fails with an unresolved column:
SELECT k,
(SELECT MAX(rn) FROM (SELECT ROW_NUMBER() OVER (ORDER BY x) AS rn
FROM VALUES (1) AS q(x))) AS sq
FROM VALUES ('drop', 2), ('keep', 1) AS input(k, v)
GROUP BY ROLLUP(k)
HAVING k = 'keep'
ORDER BY SUM(v)Such queries also lose the UpdateOuterReferences rewrite of outer aggregate references in HAVING subqueries, and they reach the command failure this review raises separately.
When the HAVING reads such an alias, for example HAVING sq = 1 AND k = 'keep' next to COUNT(*) OVER () AS group_count, the condition counts as reading a window output and stays above every real window, so group_count would still be 3 instead of the 1 that GROUP BY k gives, although the migration note and the config doc say such conditions now run first.
This extends an existing discussion.
Verification:
- Regression: By default, the ROLLUP query whose only window is inside a SELECT-list subquery, with HAVING k = 'keep' and ORDER BY SUM(v), analyzes and returns the keep row with the filter directly over the aggregate.
- Regression: By default, the ROLLUP query with COUNT(*) OVER () AS group_count and HAVING sq = 1 AND k = 'keep', where sq is a windowed subquery alias, returns group_count 1, as GROUP BY k does, and with the legacy flag it returns the base-branch value 3.
- Compatibility: A grouping-analytics HAVING that reads a real SELECT-list window alias keeps the filter above every window under both flag values.
There was a problem hiding this comment.
Fixed in b970dec. Window-output classification now uses expression-only traversal (expr.exists(_.isInstanceOf[WindowExpression])), which does not descend into subquery plans. Added both requested regressions: a subquery-only window with ORDER BY SUM(v), and a subquery alias alongside a real window with default/legacy results.
| COUNT(*) OVER () AS group_count | ||
| FROM VALUES ('drop', 2), ('keep', 1) AS input(k, v) | ||
| GROUP BY ROLLUP(k) | ||
| HAVING SUM(v) = 1L |
There was a problem hiding this comment.
Nit (P3): With 1L this golden still has the same analyzed plan by default and with the legacy flag: Window over Project [k] over Filter (sum(v) = 1) over the Aggregate, which is the older route. When tryResolveHavingCondition first sees this condition, v only becomes a TempResolvedColumn inside that method and sum is resolved later in the same pass, so the condition is unresolved and the method returns early; by then the ROLLUP aggregate has been built, and the HAVING goes through ExtractWindowExpressions and ResolveAggregateFunctions instead. No other golden adds a HAVING-only aggregate on this route, so the Project that drops HAVING-only aggregates above DeferredHaving never runs in a test, and a regression there would go unnoticed. A HAVING-only aggregate whose arguments are literals, such as HAVING COUNT(*) = 1L, is already resolved at that point and would reach it.
This extends an existing discussion.
Verification:
- Regression: A grouping-analytics query whose HAVING reads only an aggregate that is absent from the SELECT list and already resolved before the aggregate is built has different default and legacy plans and results: by default the window counts only the surviving groups and the output omits the HAVING-only aggregate.
There was a problem hiding this comment.
Fixed in b970dec. The HAVING-only aggregate case now uses HAVING COUNT(*) = 1L with data that distinguishes the surviving groups, so it reaches the deferred branch and exercises the projection that drops the extra aggregate. Default and legacy goldens differ as expected.
| HAVING k = 'keep' | ||
| ORDER BY k; | ||
|
|
||
| -- Lateral aliases must be resolved before grouping analytics HAVING is deferred. |
There was a problem hiding this comment.
Nit (P3): These cases use one alias level (SUM(v) AS s, s * 2 AS s2) and a HAVING on k. resolveLateralColumnAliases converges for them after two applications of ResolveLateralColumnAliasReference, while a chained alias such as s2 + 1 AS s3 needs a third, because the first unwrap leaves the reference to s2 for a later round. And because the condition reads k, the placement walk always ends at the Aggregate, so its terminal case on a lifted-alias Project, and the Project and Generate branches of hasWindowInWindowChain, never run. By the review's reading of the code, a lift capped at two rounds, or a terminal accepted only on an Aggregate, would leave every new golden unchanged, yet a query that adds s2 + 1 AS s3 with HAVING k = 'keep', or one with HAVING s = 1L, would fall back to HAVING above the window and count the removed groups (group_count 3 instead of 1). The literal must be 1L: with an int literal the condition needs a cast and takes the older route.
Verification:
- Regression: A grouping-analytics query whose lateral aliases chain through three levels evaluates a grouping-key HAVING below the window by default and counts only the surviving groups.
- Regression: A grouping-analytics query with lateral aliases whose HAVING reads a lifted aggregate alias places the filter directly above the Project that defines the alias, below the window, by default, and keeps it above the window with the legacy flag.
There was a problem hiding this comment.
Fixed in b970dec. Added a third lateral-alias level (s3) to exercise fixed-point lifting and a separate HAVING s = 1L case to exercise placement on the lifted-alias Project. Both default and legacy plans/results are covered.
| -- Nested windows over an aggregate, and in QUALIFY together with HAVING. | ||
| SELECT col1, MAX(RANK() OVER (ORDER BY SUM(col2))) OVER () AS m FROM t1 GROUP BY col1; | ||
|
|
||
| -- HAVING must be evaluated before every nested window layer. |
There was a problem hiding this comment.
Nit (P3): This block starts between the two queries of the existing section headed -- Nested windows over an aggregate, and in QUALIFY together with HAVING. at line 100. The QUALIFY-with-HAVING query that comment describes, which directly followed the section's first query on the base branch, now sits after RESET spark.sql.legacy.evaluateHavingAfterInnerWindow at line 323 with no comment of its own, and the regenerated analyzer and result goldens carry the same order. Moving the new tests after that query, in their own section, and regenerating the goldens restores it.
Verification:
- Inspection: The section comment about nested windows and QUALIFY with HAVING is followed by both of its queries, and the regenerated golden files differ from the current ones only in order.
There was a problem hiding this comment.
Fixed in b970dec. Restored the existing QUALIFY-with-HAVING query directly under its section header, moved the new HAVING cases after that section, and regenerated the execution and analyzer goldens.
### What changes were proposed in this pull request? Following on to #59829, refactor the scheduled build status updater (`update_build_status.yml`) completely: - Use batched GraphQL requests instead of per-PR REST requests. - Only update PR checks that need to be changed. Do a full sync periodically to catch build updates not associated with PR updates (e.g. a job is rerun with no new commit). - Clean up the log to only show key metrics and updates made. - Move the core logic into its own JavaScript file next to the workflow. Example of cleaned up log: ```text #60090 1dac863: queued -> in_progress #59954 48a5471: in_progress -> success 726 PRs checked, 36 skipped. Build checks: 7 updated, 2 created, 681 unchanged. ``` ### Why are the changes needed? Currently, an update pass takes about 10 minutes on a 15-minute schedule. That slows down status updates and makes changes to the workflow painful to test. Nearly all of that time goes to per-PR REST reads for PRs that haven't changed in weeks, and to rewriting checks that haven't changed. Fork runs usually finish within a few hours of a push. Of the latest fork runs on 690 open PRs, 8 were rerun more than 3 days after the push. The full sync picks those up. | | Before | After | |---|---|---| | Time per pass | ~10 min | ~45-60 s (~3-5 min for a full sync) | | PRs synced per pass | ~725 (all open) | ~170 | | API reads per pass | ~1,480 REST | 7 GraphQL | | Check updates per pass | ~685 | ~3 | | Log lines per pass | ~3,400 | ~10-15 | | Runner time per day | ~16 h | ~3 h | ### Does this PR introduce _any_ user-facing change? No, not for end-users. For contributors: `Build` checks update the same way, just sooner. A fork run rerun more than 3 days after the PR was last updated shows up on the next full sync, which is scheduled every 4 hours, or on the next pass after any PR activity (e.g. a comment). Contributor guidance now explains enabling Actions, using a non-master branch, and retriggering builds. To keep fork master as an upstream-sync branch, new Build checks for PRs from master provide `action_required` guidance rather than linking to a build, even if an older workflow ran there; existing run-linked checks continue updating. ### How was this patch tested? Via an agent: I replayed the old and new scripts read-only against live `apache/spark` PRs, intercepting writes. The new script made no write the old one didn't. All but one of the updates it skipped were no-ops. The remaining one was a stuck rerun on a PR outside the 3-day window, which the full sync covers. A full-sync replay synced all 727 open PRs. I also replayed both scripts with existing checks hidden, so every check had to be created. 724 of 725 creates matched, and each new check had its fork run's state. The one difference was a PR pushed a few minutes earlier with no fork run yet, which the new script left for the next pass. I ran quick and full syncs locally through `actions/github-script` against live PR data, intercepting writes and simulating responses, including ignored rerun updates. They checked 169 and 729 PRs in 57 seconds and 264 seconds, and logged 7 and 8 lines. All 26 focused tests passed, covering unchanged checks, applied and ignored updates, new checks, PRs from a fork's `master`, and errors. The test and dry-run harnesses load the JavaScript module directly. Replaying the 3 open PRs from a fork's `master` with their checks hidden, each got `action_required`. Tested on GitHub against my personal fork with a real `GITHUB_TOKEN`: a full read-only sync checked 728 PRs in 4m30s using 18 GraphQL points. A temporary fork PR verified check creation, updating the same check from queued to cancelled, and zero API writes on an unchanged repeat, with no duplicate checks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot Closes #60098 from nchammas/build-status-optimization. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> skip-checks: true
dade86c to
b970dec
Compare
What changes were proposed in this pull request?
This change places
HAVINGbelow every nested SELECT-listWindowas an all-or-noneoperation. The window-chain walk is bounded to its
Project/Generate/Windowspine,and window-output classification inspects expression trees without traversing subquery plans.
For grouping analytics (
ROLLUP,CUBE, andGROUPING SETS), an unresolvedDeferredHavingmarker preserves the aggregate output only while window extraction builds thewindow chain. The marker is converted to the placed
Filterimmediately afterExtractWindowExpressionsin the main resolution batch, so QUALIFY resolution,nondeterministic-expression handling, commands, and later analyzer batches see the normal plan
shape. Lateral column aliases are lifted to a fixed point before deferral. Conditions that
reference a window output, use a grouping function, or cannot complete the lateral-alias lift
retain their previous analysis path.
It adds the internal session config
spark.sql.legacy.evaluateHavingAfterInnerWindow, defaulting tofalse, to restore theprevious behavior when needed. The SQL migration guide documents the correction, grouping
analytics behavior, and fallback.
SQL execution and analyzer goldens cover nested window specifications, three nesting levels,
mixed nesting depths, generators over windowed inputs and subqueries, lateral aliases, grouping
functions, HAVING-only aggregates, scalar subqueries, QUALIFY, seeded nondeterministic predicates,
and view/cache/CTAS/RTAS command paths. The regenerated goldens and command outputs show
HAVINGbelow the complete SELECT-list window chain by default, while the legacy configurationretains the prior plans and results.
Why are the changes needed?
The fixed-point analyzer previously extracted one window layer per iteration and immediately
converted
HAVINGto aFilter. A later iteration could insert an innerWindowbelow thatfilter. Generator extraction produced a similar non-contiguous window chain, and grouping
analytics followed a separate path that could place the filter after a window. In each case, a
window function could see groups that
HAVINGshould have removed and produce an incorrectresult.
See SPARK-60873.
Does this PR introduce any user-facing change?
Yes. Nested window functions in grouped queries, including grouping analytics, are now evaluated
after
HAVINGwhen the condition does not depend on a window output or grouping function. Forthe regression query, the surviving
keepgroup previously producedrank_sum = 2; it nowcorrectly produces
rank_sum = 1because the removed group is excluded from every window layer.Set
spark.sql.legacy.evaluateHavingAfterInnerWindowtotrueto restore the previousbehavior.
How was this patch tested?
Added SQL execution and analyzer golden coverage. On
b970dec8938b:window-resolution-edge-cases.sql,generators-resolution-edge-cases.sql, andwindow-with-aggregates.sqlpassed (6/6).git diff --check, changed-Scala line-length, and non-ASCII checks passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)