feat: reuse range partitioning for audited date_bin aggregation by benbellick · Pull Request #26835 · apache/datafusion

benbellick · 2026-10-27T21:41:27Z

Which issue does this PR close?

This completes the initial single-key work alongside #26449. Explicit origins (#26913) and compound range keys (#24644) remain follow-ups.

Rationale for this change

Extend the aggregate-locality optimization from #26449 to date_bin. When bins stay within timestamp range partitions, aggregation can skip the hash shuffle.

What changes are included in this PR?

Pass literal argument values to the existing admission hook. INTERVAL '1 hour' and INTERVAL '1 month' have the same IntervalMonthDayNano type, so types alone cannot distinguish them. Inspecting the literal value lets date_bin require a positive fixed-duration stride without months, treating each day as 24 hours. date_trunc continues to rely only on argument types.

Initially, admit only two-argument calls over timezone-less nanosecond timestamps, using that literal stride. Explicit origins and other types remain excluded.

For example:

SELECT
    date_bin(INTERVAL '1 hour', event_time) AS hour_bin,
    COUNT(*)
FROM events
GROUP BY hour_bin;

For ascending timestamp ranges:

A split at 01:00 separates hourly bins, so the shuffle can be skipped.
A split at 01:30 cuts through the 01:00 bin, so the shuffle remains.

Every split must pass the existing boundary proof. Evaluation errors conservatively retain the shuffle. This follows the NULL-to-error fix in #26607.

What is the testing strategy for this PR?

Unit tests cover admission, hook forwarding, ascending/descending boundaries, and evaluation errors. SQLLogicTests verify plans and results for aligned bins, crossing bins, and excluded explicit origins.

Are there any user-facing changes?

Eligible date_bin aggregations can avoid a hash shuffle. Function results are unchanged.

Note: This PR was developed with AI assistance. All changes have been reviewed, and I take full responsibility for this contribution.

benbellick · 2026-10-28T18:50:39Z

cc @NGA-TRAN @jayzhan211 This is extending the previous work on date_trunc to also work on (a narrow case of) date_bin. Thanks!

NGA-TRAN

Looks great. I just have a minor comment. Thanks @benbellick

NGA-TRAN · 2026-10-29T00:17:37Z

-06)----------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-1.parquet]]}, projection=[key, col4, timestamp, value], output_ordering=[key@0 ASC, timestamp@2 ASC], output_partitioning=Range([timestamp@2 ASC], [(1704070800000000000)], 2), file_type=parquet, predicate=col4@4 = a, pruning_predicate=col4_null_count@2 != row_count@3 AND col4_min@0 <= a AND a <= col4_max@1, required_guarantees=[col4 in (a)]
+02)--AggregateExec: mode=SinglePartitioned, gby=[key@0 as key, date_bin(IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }, timestamp@1) as date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)], aggr=[sum(range_sorted_time_bin.value)], group_clustering_mode=Full
+03)----FilterExec: col4@1 = a, projection=[key@0, timestamp@2, value@3]
+04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-1.parquet]]}, projection=[key, col4, timestamp, value], output_ordering=[key@0 ASC, timestamp@2 ASC], output_partitioning=Range([timestamp@2 ASC], [(1704070800000000000)], 2), file_type=parquet, predicate=col4@4 = a, pruning_predicate=col4_null_count@2 != row_count@3 AND col4_min@0 <= a AND a <= col4_max@1, required_guarantees=[col4 in (a)]


NGA-TRAN · 2026-10-29T00:30:25Z

+# An explicit origin remains outside the audited slice, even when it is
+# the epoch and produces the same bins as the two-argument expression.
+##########


Do do think this comment will be clearer?

Suggested change

# An explicit origin remains outside the audited slice, even when it is

# the epoch and produces the same bins as the two-argument expression.

##########

# This PR enables shuffle removal only for two-argument date_bin calls.

# A three-argument call therefore retains the shuffle, even when its

# explicit epoch origin produces identical bins and would be safe.

##########

Do you plan to support this sometime; or maybe open a ticket for it?

I opened a ticket for it and added a reference to it in the comment. Thanks!

#26913

jayzhan211

@benbellick thanks for extending this to date_bin! The admission logic looks right to me. My main ask is inline: could you fold the literal arguments into the existing, still-unreleased hook instead of adding a second one? The other inline comment is optional.

github-actions · 2026-10-29T15:08:52Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning 676c003146e2dda8f3aab3c7401e6c6da61cc131
    Building datafusion-expr v55.1.0 (current)
       Built [  26.690s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.096s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [   6.335s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.096s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   2.207s] 229 checks: 227 pass, 2 fail, 0 warn, 31 skip

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_expr::ScalarUDF::supports_range_key_locality_analysis takes 1 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-676c003146e2dda8f3aab3c7401e6c6da61cc131/e7d6629c5a2b043ae779cce0fa31961044e750d3/datafusion/expr/src/udf.rs:355, but now takes 2 parameters in /home/runner/work/datafusion/datafusion/datafusion/expr/src/udf.rs:355

--- failure trait_method_parameter_count_changed: pub trait method parameter count changed ---

Description:
A trait method now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/trait_method_parameter_count_changed.ron

Failed in:
  ScalarUDFImpl::supports_range_key_locality_analysis now takes 2 instead of 1 parameters, in file /home/runner/work/datafusion/datafusion/datafusion/expr/src/udf.rs:1006

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  37.198s] datafusion-expr
    Building datafusion-functions v55.1.0 (current)
       Built [  21.452s] (current)
     Parsing datafusion-functions v55.1.0 (current)
      Parsed [   0.093s] (current)
    Building datafusion-functions v55.1.0 (baseline)
       Built [   6.070s] (baseline)
     Parsing datafusion-functions v55.1.0 (baseline)
      Parsed [   0.094s] (baseline)
    Checking datafusion-functions v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.465s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  29.653s] datafusion-functions
    Building datafusion-physical-expr v55.1.0 (current)
       Built [   3.196s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.052s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [   3.174s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.425s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [   7.516s] datafusion-physical-expr

benbellick · 2026-10-30T02:57:17Z

@jayzhan211 Thanks for the review :) just updated everything!

jayzhan211 · 2026-10-31T02:10:42Z

Thanks @benbellick @NGA-TRAN !

benbellick added 3 commits October 23, 2026 18:43

refactor(expr): pass literal arguments to range-locality audits

f9defc3

feat: reuse range partitioning for audited date_bin aggregation

ab2c274

test(datetime): remove redundant date_bin correctness coverage

a9319fa

github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Oct 27, 2026

benbellick added 7 commits October 27, 2026 21:46

test(datetime): split date_bin locality admission checks

ddc260e

test(physical-expr): clarify literal locality audit coverage

6764580

test: trim redundant range-locality admission checks

eb9b139

test: clarify date_bin range boundary scenarios

ca5d25f

docs: clarify date_bin test expression helper

5de4293

test(physical-expr): remove optional null-literal metadata check

246b515

test(physical-expr): use date_bin for locality hook coverage

fd93941

benbellick marked this pull request as ready for review October 28, 2026 18:47

docs: clarify date_bin locality test helper

3f8f71b

NGA-TRAN approved these changes Oct 29, 2026

View reviewed changes

jayzhan211 self-requested a review October 29, 2026 13:08

jayzhan211 requested changes Oct 29, 2026

View reviewed changes

Comment thread datafusion/expr/src/udf.rs Resolved

Comment thread datafusion/functions/src/datetime/date_bin.rs Resolved

refactor(expr): consolidate range locality audit hooks

d3dbb82

github-actions Bot added the auto detected api change Auto detected API change label Oct 29, 2026

feat(datetime): admit fixed day strides for range locality

890fffd

This was referenced Oct 30, 2026

Support range-key locality analysis for date_bin with a literal origin #26913

Closed

Support functions date_bin and date_trunc in range partition satisfaction #23569

Open

docs: clarify date_bin explicit-origin follow-up

31d3ee4

benbellick requested a review from jayzhan211 October 30, 2026 02:57

jayzhan211 approved these changes Oct 31, 2026

View reviewed changes

jayzhan211 added this pull request to the merge queue Oct 31, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026

jayzhan211 added this pull request to the merge queue Oct 31, 2026

Merged via the queue into apache:main with commit f1ecc4a Oct 31, 2026
44 checks passed

github-actions Bot added the v56.0.0 label Oct 31, 2026

This was referenced Nov 10, 2026

feat: skip hash shuffle for date_bin/date_trunc on Range([timestamp]) #24501

Closed

Support date_bin / date_trunc satisfaction for multi-key Range partitioning #24644

Open

Navigation Menu

Repository navigation

feat: reuse range partitioning for audited date_bin aggregation - #26835

feat: reuse range partitioning for audited date_bin aggregation#26835
jayzhan211 merged 14 commits into
apache:mainfrom
benbellick:refactor/range-locality-literal-arguments

benbellick commented Oct 27, 2026 •

edited

Loading

Uh oh!

benbellick commented Oct 28, 2026

Uh oh!

NGA-TRAN left a comment

Uh oh!

NGA-TRAN Oct 29, 2026

Uh oh!

NGA-TRAN Oct 29, 2026

Uh oh!

benbellick Oct 30, 2026

Uh oh!

jayzhan211 left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

benbellick commented Oct 30, 2026

Uh oh!

jayzhan211 commented Oct 31, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

benbellick commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

benbellick commented Oct 28, 2026

Uh oh!

NGA-TRAN left a comment

Choose a reason for hiding this comment

Uh oh!

NGA-TRAN Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

NGA-TRAN Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

benbellick Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benbellick commented Oct 30, 2026

Uh oh!

jayzhan211 commented Oct 31, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benbellick commented Oct 27, 2026 •

edited

Loading

github-actions Bot commented Oct 29, 2026 •

edited

Loading