Skip to content

fix: wrong aggregate functional dependencies when a GROUP BY column is not determined by the key - #27113

Merged
jayzhan211 merged 4 commits into
apache:mainfrom
asolimando:asolimando/aggregate-fd-single-coverage
Nov 11, 2026
Merged

jayzhan211 merged 4 commits into
apache:mainfrom
asolimando:asolimando/aggregate-fd-single-coverage

Conversation

@asolimando

@asolimando asolimando commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

The subquery below groups by i_item_sk, s_date. Item 1 has two dates, so it forms two groups:

SELECT sum(o_amount) FROM orders
JOIN (SELECT i_item_sk, s_date FROM item JOIN sales ON i_item_sk = s_item_sk
      GROUP BY i_item_sk, s_date) f
  ON o_item_sk = f.i_item_sk;
-- 30, while the correct result is 40 (see #27112 for the data)

aggregate_functional_dependencies marked i_item_sk as Dependency::Single on the output of f (a key: at most one row per value). That is wrong, because item 1 has two rows.

It did so because nothing determines s_date. The check compared what i_item_sk determines with what the whole GROUP BY list determines. s_date was in neither set, so the two sets were equal.

EliminateJoin trusted the wrong key and turned the join into a semi join (each orders row kept once instead of once per matching group). Item 1's order was counted once, hence 30.

The function also gave every dependency all output columns as target_indices. That claims i_item_sk determines s_date, which breaks other rules, for example an outer GROUP BY i_item_sk, s_date drops s_date.

What changes are included in this PR?

  • A re-expressed dependency is now Single only if it is Single in the input or its determinant determines every GROUP BY expression. A deterministic computed GROUP BY expression is determined when every column it reads is, for example upper(i_desc) by i_item_sk. Volatile expressions, and expressions with subqueries or outer references, are never determined.
  • Otherwise the dependency keeps its mode and targets only the GROUP BY expressions it determines, not the aggregate values.
  • The dependency on the whole GROUP BY list is skipped only when a Single dependency replaces it, and its extraneous expressions are removed: columns that the other GROUP BY columns determine, and deterministic computed expressions whose fields the remaining key columns determine. The SQL planner adds determined columns to the GROUP BY, so without this the dependency was lost when a projection removed them, and selecting an aggregate value through such a projection failed to plan.
  • aggregate_functional_dependencies keeps its signature and is deprecated. The new aggregate_functional_dependencies_with_computed_inputs takes the columns that each computed GROUP BY expression reads.

What is the testing strategy for this PR?

SQL logic tests in functional_dependencies.slt:

  • the query above (the inner join is kept)
  • GROUP BY the key alone (the semi join is still used)
  • an outer GROUP BY over the aggregate keeps all groups
  • an aggregate value selected through a projection that removes a determined column
  • the DISTINCT and window-frame symptoms from the issue (the DISTINCT is kept, the frame stays RANGE)
  • a deterministic computed expression next to the key (upper(i_desc)), one that reads an undetermined column (s_date + 1), and a volatile one (random() > 0.5)
  • computed expressions next to a column the key does not determine (GROUP BY i_item_sk, s_date, upper(i_desc), s_date + 1), and a GROUP BY that reads no column (GROUP BY 'x')

The first and third fail without the fix, and the projection case fails without the change to the GROUP BY dependency. The s_date + 1 case fails if every computed expression counts as determined, and the random() case fails without the volatility check. The GROUP BY i_item_sk, s_date, ... case fails without removing computed expressions from the GROUP BY key, and the GROUP BY 'x' case returns no row instead of 0 if a key can have no columns. All sqllogictests pass.

Are there any user-facing changes?

Yes. Queries such as the one above now return the correct result. Queries that relied on the wrong dependencies, such as SELECT s_date ... GROUP BY i_item_sk over f, are now rejected at planning.

aggregate_functional_dependencies is deprecated in favour of aggregate_functional_dependencies_with_computed_inputs, as described in the upgrade guide. There are no breaking API changes.


Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

…s not determined by the key

`aggregate_functional_dependencies` marked a key as `Single` on the
aggregate output when another GROUP BY column was determined by no input
dependency, and gave every re-expressed dependency all output columns as
targets. Rules that trust these dependencies, such as `EliminateJoin`,
returned wrong results.

Closes apache#27112.
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Nov 5, 2026
@asolimando
asolimando marked this pull request as ready for review November 5, 2026 08:53
@xudong963
xudong963 self-requested a review November 6, 2026 14:04

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asolimando thanks for the fix! It resolves every symptom listed in #27112. I'm requesting changes for one regression: some queries that return correct results on 55.2.0 now fail to plan. The details and a tested fix are in the inline comment.

… by a key

A computed GROUP BY expression was never counted as determined, so a key
next to one, such as `GROUP BY i_item_sk, upper(i_desc)`, was no longer a
key of the aggregate output, and queries that plan on 55.2.0 failed.

The new `aggregate_functional_dependencies_with_computed_inputs` takes a
`computed_input_indices` parameter with the input fields each
deterministic computed GROUP BY expression reads. Such an expression is
determined when every field it reads is. Volatile expressions and
expressions with subqueries or outer references are never determined.
`aggregate_functional_dependencies` keeps its signature and is deprecated
since 57.0.0; it treats every computed expression as not determined.

Also adds tests for the DISTINCT and window frame symptoms of apache#27112.
@github-actions github-actions Bot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions labels Nov 8, 2026
@asolimando
asolimando requested a review from jayzhan211 November 8, 2026 22:08

@xudong963 xudong963 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @asolimando

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asolimando thanks for adding the computed-expression support so quickly. One more shape still fails to plan although it works on 55.2.0; details and a tested fix inline.

Comment on lines +580 to +589
let source_indices = match get_required_group_by_exprs_indices(
aggr_input_schema,
group_by_input_indices,
) {
Some(mut indices) => {
indices.sort_unstable();
indices
}
None => (0..count).collect(),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asolimando thanks for the update! This still rejects a query that returns correct results on 55.2.0 and main, when the computed expression sits next to a column the key doesn't determine. The query below returns 3 rows on 55.2.0 and fails to plan with this PR. My earlier suggestion missed this path: only the whole GROUP BY key can prove (i_item_sk, s_date) is a key here, and get_required_group_by_exprs_indices gives up on any computed expression. The change below drops a computed expression from that key when the remaining key columns determine every field it reads. With it, the full sqllogictest suite passes.

SELECT i_item_sk, s_date, c
FROM (SELECT i_item_sk, s_date, count(*) AS c FROM item JOIN sales ON i_item_sk = s_item_sk
      GROUP BY i_item_sk, s_date, upper(i_desc)) q
GROUP BY i_item_sk, s_date;
Change in functional_dependencies.rs
@@ -577,16 +577,11 @@ pub fn aggregate_functional_dependencies_with_computed_inputs(
         // Remove extraneous columns (the GROUP BY expressions that the others
         // determine) from the key, so that the key still exists after a
         // projection removes them.
-        let source_indices = match get_required_group_by_exprs_indices(
+        let source_indices = group_by_key_indices(
             aggr_input_schema,
             group_by_input_indices,
-        ) {
-            Some(mut indices) => {
-                indices.sort_unstable();
-                indices
-            }
-            None => (0..count).collect(),
-        };
+            computed_input_indices,
+        );
@@ -617,6 +612,58 @@ pub fn aggregate_functional_dependencies_with_computed_inputs(
     FunctionalDependencies::new(aggregate_func_dependencies)
 }
 
+/// Returns the GROUP BY expressions that the whole GROUP BY key needs. A column
+/// that the other columns determine is left out, and so is a deterministic
+/// computed expression whose fields the remaining columns determine.
+fn group_by_key_indices(
+    schema: &DFSchema,
+    group_by_input_indices: &[Option<usize>],
+    computed_input_indices: &[Option<Vec<usize>>],
+) -> Vec<usize> {
+    let all = || (0..group_by_input_indices.len()).collect();
+    let (column_positions, column_inputs): (Vec<usize>, Vec<Option<usize>>) =
+        group_by_input_indices
+            .iter()
+            .enumerate()
+            .filter(|(_, input_idx)| input_idx.is_some())
+            .map(|(idx, input_idx)| (idx, *input_idx))
+            .unzip();
+    let Some(required) = get_required_group_by_exprs_indices(schema, &column_inputs)
+    else {
+        return all();
+    };
+    let key_inputs = required
+        .iter()
+        .map(|idx| column_inputs[*idx])
+        .collect::<Vec<_>>();
+    let determined =
+        get_target_functional_dependencies(schema, &key_inputs).unwrap_or_default();
+    let mut key = required
+        .iter()
+        .map(|idx| column_positions[*idx])
+        .collect::<Vec<_>>();
+    for (idx, (input_idx, read)) in group_by_input_indices
+        .iter()
+        .zip(computed_input_indices)
+        .enumerate()
+    {
+        let is_determined = read.as_ref().is_some_and(|read| {
+            read.iter().all(|field| {
+                determined.contains(field) || key_inputs.contains(&Some(*field))
+            })
+        });
+        if input_idx.is_none() && !is_determined {
+            key.push(idx);
+        }
+    }
+    // A key with no columns is left as the whole GROUP BY list.
+    if key.is_empty() {
+        return all();
+    }
+    key.sort_unstable();
+    key
+}
+
Tests to add after 8.5 (the first fails on this PR; the second fails without the empty-key fallback, where count(*) returns no rows)
# `upper(i_desc)` reads `i_desc`, which `i_item_sk` determines, and `s_date + 1`
# reads `s_date`: both are removed from the key of the inner aggregate.
query III
SELECT i_item_sk, s_date, c
FROM (SELECT i_item_sk, s_date, count(*) AS c FROM item JOIN sales ON i_item_sk = s_item_sk
      GROUP BY i_item_sk, s_date, upper(i_desc), s_date + 1) q
GROUP BY i_item_sk, s_date ORDER BY i_item_sk, s_date;
----
1 100 1
1 101 1
2 100 1

# A GROUP BY that reads no column keeps the whole GROUP BY list as its key.
query I
SELECT count(*) FROM orders
JOIN (SELECT 1 AS k FROM sales WHERE false GROUP BY 'x') g ON o_item_sk = g.k;
----
0

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch! I addressed your comments with aef8312, which applies your change and adds your two tests as 8.6 and 8.7. I checked that 8.6 fails without the change, and that 8.7 returns no row without the empty-key fallback.

On top of it, I did a pass in 360c072 for readability: deciding whether a GROUP BY expression is determined is now done in one place, the new group_by_positions_determined_by helper, which both the dependency loop and group_by_key_indices use. I also gave both helpers a doc with an example, as the logic is quite complex. The behavior is the same, and the tests are unchanged. I kept it as a separate commit, so we can easily drop it if you prefer your version.

…Y key

`get_required_group_by_exprs_indices` gives up on any computed GROUP BY
expression, so the GROUP BY key kept every expression. A projection that
removed one of them dropped the key, and a query such as
`GROUP BY i_item_sk, s_date, upper(i_desc)` below an outer
`GROUP BY i_item_sk, s_date` failed to plan, although it plans on 55.2.0.

The new `group_by_key_indices` removes extraneous columns from the key
as before, and also a deterministic computed expression whose fields the
remaining key columns determine. A key left with no columns falls back to
the whole GROUP BY list.
The dependency loop and the GROUP BY key reduction each checked whether
a GROUP BY expression is determined, in two different ways. Both now use
`group_by_positions_determined_by`, which returns the GROUP BY positions
that a set of input fields determines. `group_by_key_indices` builds that
set from the GROUP BY columns and what the remaining key columns
determine, and keeps the key columns and every expression outside it.
@asolimando
asolimando requested a review from jayzhan211 November 9, 2026 23:13
@jayzhan211
jayzhan211 added this pull request to the merge queue Nov 11, 2026
@jayzhan211

Copy link
Copy Markdown
Contributor

Thanks @asolimando @xudong963 !

Merged via the queue into apache:main with commit 21892ed Nov 11, 2026
46 checks passed
@asolimando
asolimando deleted the asolimando/aggregate-fd-single-coverage branch November 12, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong results: an aggregate over a primary key and another column is treated as unique on the key

3 participants