Skip to content

[codex POC] Add query fusion optimizer - #23175

Closed
nathanb9 wants to merge 2 commits into
apache:mainfrom
nathanb9:codex/query-fusion-poc
Closed

[codex POC] Add query fusion optimizer#23175
nathanb9 wants to merge 2 commits into
apache:mainfrom
nathanb9:codex/query-fusion-poc

Conversation

@nathanb9

@nathanb9 nathanb9 commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

Draft POC for experimenting with Athena-style computation reuse in DataFusion.

What changes are included in this PR?

Adds a disabled-by-default query_fusion optimizer rule with narrow POC rewrites for:

  • UNION ALL branches over the same source using branch tags.
  • Scalar aggregates over the same source using one aggregate with filtered aggregate expressions.
  • Direct source JOIN Aggregate(source GROUP BY keys) shapes using window aggregates.

Are these changes tested?

  • cargo test -p datafusion-optimizer query_fusion --lib
  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • Verified TP-DS Q9 optimized plan uses one fused scalar aggregate when enabled.

Are there any user-facing changes?

No default behavior change. The POC is gated behind datafusion.optimizer.enable_query_fusion.

@github-actions github-actions Bot added optimizer Optimizer rules common Related to common crate physical-plan Changes to the physical-plan crate labels Jun 25, 2026
@nathanb9 nathanb9 changed the title [codex] Add query fusion optimizer POC [codex POC] Add query fusion optimizer Jun 25, 2026
@nathanb9
nathanb9 force-pushed the codex/query-fusion-poc branch from 691a3c7 to da280f8 Compare June 25, 2026 03:59
@github-actions github-actions Bot removed the physical-plan Changes to the physical-plan crate label Jun 25, 2026
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion-common v54.0.0 (current)
       Built [  32.725s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.063s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  31.995s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.064s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.037s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field OptimizerOptions.enable_query_fusion in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1347

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  67.350s] datafusion-common
    Building datafusion-optimizer v54.0.0 (current)
       Built [  25.802s] (current)
     Parsing datafusion-optimizer v54.0.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-optimizer v54.0.0 (baseline)
       Built [  25.915s] (baseline)
     Parsing datafusion-optimizer v54.0.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.238s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  52.801s] datafusion-optimizer

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jun 25, 2026
@nathanb9

nathanb9 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Query / Computation Fusion Optimizer Ideas

Inspired by Athena's computation-reuse fusion paper.

1. Scalar Aggregate Fusion

Merge multiple scalar aggregate subqueries over the same source into one aggregate with FILTER clauses.

Example query: TPC-DS Q9

Before:

SELECT
  (SELECT avg(x) FROM R WHERE p1),
  (SELECT avg(y) FROM R WHERE p2);

After:

SELECT
  avg(x) FILTER (WHERE p1),
  avg(y) FILTER (WHERE p2)
FROM R;

2. Grouped Aggregate Join To Window Aggregate

Replace a join back to a grouped aggregate over the same source with a window aggregate.

Example query: TPC-DS Q65

Before:

SELECT R.*, A.avg_v
FROM R
JOIN (
  SELECT k, avg(v) AS avg_v
  FROM R
  GROUP BY k
) A
ON R.k = A.k;

After:

SELECT
  R.*,
  avg(v) OVER (PARTITION BY k) AS avg_v
FROM R
WHERE k IS NOT NULL;

3. UNION ALL Branch Fusion

Merge same-source UNION ALL branches into one scan plus branch tags and branch-specific predicates.

Example query: TPC-DS Q23-style shapes

Before:

SELECT ... FROM R WHERE p1
UNION ALL
SELECT ... FROM R WHERE p2;

After:

SELECT ...
FROM R
CROSS JOIN branches
WHERE (branch = 1 AND p1)
   OR (branch = 2 AND p2);

4. Multi-Aggregate Common-Source Fusion

Merge sibling aggregates over the same source and group keys into one aggregate with multiple aggregate expressions.

Example queries: TPC-DS Q28 / Q88

Before:

SELECT ...
FROM (
  SELECT k, agg1(v) FROM R GROUP BY k
) a,
(
  SELECT k, agg2(v) FROM R GROUP BY k
) b;

After:

SELECT
  k,
  agg1(v),
  agg2(v)
FROM R
GROUP BY k;

Note: Q28-style cases may need DISTINCT / MarkDistinct handling.

5. Common Join-Input Fusion

Detect joins or subqueries that repeatedly consume the same source and share the common input instead of recomputing it.

Example queries: TPC-DS Q64 / Q88

Before:

SELECT ...
FROM (R JOIN A) x,
     (R JOIN B) y;

After:

SELECT ...
FROM R
JOIN A
JOIN B;

This should start conservatively behind a config flag and require semantic guards for NULLs, DISTINCT, outer joins, limits, ordering, and volatile expressions.

@nathanb9 nathanb9 mentioned this pull request Aug 8, 2026
Open
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale PR has not had any activity for some time label Aug 26, 2026
@nathanb9 nathanb9 closed this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto detected api change Auto detected API change common Related to common crate optimizer Optimizer rules Stale PR has not had any activity for some time

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nathanb9