Repository navigation
fix: wrong aggregate functional dependencies when a GROUP BY column is not determined by the key - #27113
Conversation
…s not determined by the key `aggregate_functional_dependencies` marked a key as `Single` on the aggregate output when another GROUP BY column was determined by no input dependency, and gave every re-expressed dependency all output columns as targets. Rules that trust these dependencies, such as `EliminateJoin`, returned wrong results. Closes apache#27112.
jayzhan211
left a comment
There was a problem hiding this comment.
@asolimando thanks for the fix! It resolves every symptom listed in #27112. I'm requesting changes for one regression: some queries that return correct results on 55.2.0 now fail to plan. The details and a tested fix are in the inline comment.
… by a key A computed GROUP BY expression was never counted as determined, so a key next to one, such as `GROUP BY i_item_sk, upper(i_desc)`, was no longer a key of the aggregate output, and queries that plan on 55.2.0 failed. The new `aggregate_functional_dependencies_with_computed_inputs` takes a `computed_input_indices` parameter with the input fields each deterministic computed GROUP BY expression reads. Such an expression is determined when every field it reads is. Volatile expressions and expressions with subqueries or outer references are never determined. `aggregate_functional_dependencies` keeps its signature and is deprecated since 57.0.0; it treats every computed expression as not determined. Also adds tests for the DISTINCT and window frame symptoms of apache#27112.
jayzhan211
left a comment
There was a problem hiding this comment.
@asolimando thanks for adding the computed-expression support so quickly. One more shape still fails to plan although it works on 55.2.0; details and a tested fix inline.
| let source_indices = match get_required_group_by_exprs_indices( | ||
| aggr_input_schema, | ||
| group_by_input_indices, | ||
| ) { | ||
| Some(mut indices) => { | ||
| indices.sort_unstable(); | ||
| indices | ||
| } | ||
| None => (0..count).collect(), | ||
| }; |
There was a problem hiding this comment.
@asolimando thanks for the update! This still rejects a query that returns correct results on 55.2.0 and main, when the computed expression sits next to a column the key doesn't determine. The query below returns 3 rows on 55.2.0 and fails to plan with this PR. My earlier suggestion missed this path: only the whole GROUP BY key can prove (i_item_sk, s_date) is a key here, and get_required_group_by_exprs_indices gives up on any computed expression. The change below drops a computed expression from that key when the remaining key columns determine every field it reads. With it, the full sqllogictest suite passes.
SELECT i_item_sk, s_date, c
FROM (SELECT i_item_sk, s_date, count(*) AS c FROM item JOIN sales ON i_item_sk = s_item_sk
GROUP BY i_item_sk, s_date, upper(i_desc)) q
GROUP BY i_item_sk, s_date;Change in functional_dependencies.rs
@@ -577,16 +577,11 @@ pub fn aggregate_functional_dependencies_with_computed_inputs(
// Remove extraneous columns (the GROUP BY expressions that the others
// determine) from the key, so that the key still exists after a
// projection removes them.
- let source_indices = match get_required_group_by_exprs_indices(
+ let source_indices = group_by_key_indices(
aggr_input_schema,
group_by_input_indices,
- ) {
- Some(mut indices) => {
- indices.sort_unstable();
- indices
- }
- None => (0..count).collect(),
- };
+ computed_input_indices,
+ );
@@ -617,6 +612,58 @@ pub fn aggregate_functional_dependencies_with_computed_inputs(
FunctionalDependencies::new(aggregate_func_dependencies)
}
+/// Returns the GROUP BY expressions that the whole GROUP BY key needs. A column
+/// that the other columns determine is left out, and so is a deterministic
+/// computed expression whose fields the remaining columns determine.
+fn group_by_key_indices(
+ schema: &DFSchema,
+ group_by_input_indices: &[Option<usize>],
+ computed_input_indices: &[Option<Vec<usize>>],
+) -> Vec<usize> {
+ let all = || (0..group_by_input_indices.len()).collect();
+ let (column_positions, column_inputs): (Vec<usize>, Vec<Option<usize>>) =
+ group_by_input_indices
+ .iter()
+ .enumerate()
+ .filter(|(_, input_idx)| input_idx.is_some())
+ .map(|(idx, input_idx)| (idx, *input_idx))
+ .unzip();
+ let Some(required) = get_required_group_by_exprs_indices(schema, &column_inputs)
+ else {
+ return all();
+ };
+ let key_inputs = required
+ .iter()
+ .map(|idx| column_inputs[*idx])
+ .collect::<Vec<_>>();
+ let determined =
+ get_target_functional_dependencies(schema, &key_inputs).unwrap_or_default();
+ let mut key = required
+ .iter()
+ .map(|idx| column_positions[*idx])
+ .collect::<Vec<_>>();
+ for (idx, (input_idx, read)) in group_by_input_indices
+ .iter()
+ .zip(computed_input_indices)
+ .enumerate()
+ {
+ let is_determined = read.as_ref().is_some_and(|read| {
+ read.iter().all(|field| {
+ determined.contains(field) || key_inputs.contains(&Some(*field))
+ })
+ });
+ if input_idx.is_none() && !is_determined {
+ key.push(idx);
+ }
+ }
+ // A key with no columns is left as the whole GROUP BY list.
+ if key.is_empty() {
+ return all();
+ }
+ key.sort_unstable();
+ key
+}
+Tests to add after 8.5 (the first fails on this PR; the second fails without the empty-key fallback, where count(*) returns no rows)
# `upper(i_desc)` reads `i_desc`, which `i_item_sk` determines, and `s_date + 1`
# reads `s_date`: both are removed from the key of the inner aggregate.
query III
SELECT i_item_sk, s_date, c
FROM (SELECT i_item_sk, s_date, count(*) AS c FROM item JOIN sales ON i_item_sk = s_item_sk
GROUP BY i_item_sk, s_date, upper(i_desc), s_date + 1) q
GROUP BY i_item_sk, s_date ORDER BY i_item_sk, s_date;
----
1 100 1
1 101 1
2 100 1
# A GROUP BY that reads no column keeps the whole GROUP BY list as its key.
query I
SELECT count(*) FROM orders
JOIN (SELECT 1 AS k FROM sales WHERE false GROUP BY 'x') g ON o_item_sk = g.k;
----
0
There was a problem hiding this comment.
Thanks, good catch! I addressed your comments with aef8312, which applies your change and adds your two tests as 8.6 and 8.7. I checked that 8.6 fails without the change, and that 8.7 returns no row without the empty-key fallback.
On top of it, I did a pass in 360c072 for readability: deciding whether a GROUP BY expression is determined is now done in one place, the new group_by_positions_determined_by helper, which both the dependency loop and group_by_key_indices use. I also gave both helpers a doc with an example, as the logic is quite complex. The behavior is the same, and the tests are unchanged. I kept it as a separate commit, so we can easily drop it if you prefer your version.
…Y key `get_required_group_by_exprs_indices` gives up on any computed GROUP BY expression, so the GROUP BY key kept every expression. A projection that removed one of them dropped the key, and a query such as `GROUP BY i_item_sk, s_date, upper(i_desc)` below an outer `GROUP BY i_item_sk, s_date` failed to plan, although it plans on 55.2.0. The new `group_by_key_indices` removes extraneous columns from the key as before, and also a deterministic computed expression whose fields the remaining key columns determine. A key left with no columns falls back to the whole GROUP BY list.
The dependency loop and the GROUP BY key reduction each checked whether a GROUP BY expression is determined, in two different ways. Both now use `group_by_positions_determined_by`, which returns the GROUP BY positions that a set of input fields determines. `group_by_key_indices` builds that set from the GROUP BY columns and what the remaining key columns determine, and keeps the key columns and every expression outside it.
|
Thanks @asolimando @xudong963 ! |
Which issue does this PR close?
Rationale for this change
The subquery below groups by
i_item_sk, s_date. Item 1 has two dates, so it forms two groups:aggregate_functional_dependenciesmarkedi_item_skasDependency::Singleon the output off(a key: at most one row per value). That is wrong, because item 1 has two rows.It did so because nothing determines
s_date. The check compared whati_item_skdetermines with what the whole GROUP BY list determines.s_datewas in neither set, so the two sets were equal.EliminateJointrusted the wrong key and turned the join into a semi join (eachordersrow kept once instead of once per matching group). Item 1's order was counted once, hence 30.The function also gave every dependency all output columns as
target_indices. That claimsi_item_skdeterminess_date, which breaks other rules, for example an outerGROUP BY i_item_sk, s_datedropss_date.What changes are included in this PR?
Singleonly if it isSinglein the input or its determinant determines every GROUP BY expression. A deterministic computed GROUP BY expression is determined when every column it reads is, for exampleupper(i_desc)byi_item_sk. Volatile expressions, and expressions with subqueries or outer references, are never determined.Singledependency replaces it, and its extraneous expressions are removed: columns that the other GROUP BY columns determine, and deterministic computed expressions whose fields the remaining key columns determine. The SQL planner adds determined columns to the GROUP BY, so without this the dependency was lost when a projection removed them, and selecting an aggregate value through such a projection failed to plan.aggregate_functional_dependencieskeeps its signature and is deprecated. The newaggregate_functional_dependencies_with_computed_inputstakes the columns that each computed GROUP BY expression reads.What is the testing strategy for this PR?
SQL logic tests in
functional_dependencies.slt:upper(i_desc)), one that reads an undetermined column (s_date + 1), and a volatile one (random() > 0.5)GROUP BY i_item_sk, s_date, upper(i_desc), s_date + 1), and a GROUP BY that reads no column (GROUP BY 'x')The first and third fail without the fix, and the projection case fails without the change to the GROUP BY dependency. The
s_date + 1case fails if every computed expression counts as determined, and therandom()case fails without the volatility check. TheGROUP BY i_item_sk, s_date, ...case fails without removing computed expressions from the GROUP BY key, and theGROUP BY 'x'case returns no row instead of0if a key can have no columns. All sqllogictests pass.Are there any user-facing changes?
Yes. Queries such as the one above now return the correct result. Queries that relied on the wrong dependencies, such as
SELECT s_date ... GROUP BY i_item_skoverf, are now rejected at planning.aggregate_functional_dependenciesis deprecated in favour ofaggregate_functional_dependencies_with_computed_inputs, as described in the upgrade guide. There are no breaking API changes.Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.