Skip to content

Commit c48d4ed

Browse files
berkaysynnadametesynnadamustafasrepoozankabak
committed
Projection Pushdown (#211)
* [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * Fix linter errors, compile errors after rebase, Update commit hashes, regenerate proto * Rewrite Filter Predicate (#192) * Global join selection (#183) * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * Initial comments on working * License update (#157) This extension adds Synnada license information to the existing one. * Adding comments * Update sort_hash_join.rs * After merge silent error * Change the query in HashJoin * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Update rat_exclude_files.txt * Clippy solving. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * After merge * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Before clippy fmt etc. * lazy loading tables * mini test * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes * Before rebase merge * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Update utils.rs * Print deletion * Update Cargo.lock * Refactor for review * Working without slt * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Change in test folders * Update join_pipeline_selection.rs * Update utils.rs * Before clippy * Before SLT * Tests are passing and clippy OK. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <[email protected]> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * After merge * Update rat_exclude_files.txt * Comments visited * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * For mustafa * Final * Update rat_exclude_files.txt * More commenting * Fix linter errors, compile errors after rebase, Update commit hashes * After merge refactors * Dir * Additional test for coverage * Update join_disable_repartition_joins.slt * Review changes, remove code duplicates * Update subdirectory hashes --------- Co-authored-by: Berkay Şahin <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> * work in progress * fix after merge * joins are ok * code cleaning * CaseExpr handling * tests are updated * Handling sequential projections * Simplifications * partition expr update * ProjectionPushdown becomes a rule.bash de * fix after merge * remove the subrule * tpch update * Minor comment changes * remove unnecessary state struct * coalesce batches does not let to push projection down * tpch changes removed * minor changes * addresses reviews * Update projection_pushdown.rs * minor * Review Part 1 * Simplify the API of plan handlers * Review Part 2 * Review Part 3 * Review Part 4 * Review Part 5 * Review Part 6 * Review Part 7 * Remove duplication of physical_expr matching * fix documentation * Minor changes * Take upstream changes --------- Co-authored-by: Metehan Yıldırım <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: Mustafa Akur <[email protected]>
1 parent 8bd6fca commit c48d4ed

11 files changed

Lines changed: 217 additions & 130 deletions

File tree

‎datafusion/core/src/physical_optimizer/join_pipeline_selection.rs‎

Lines changed: 47 additions & 55 deletions
Large diffs are not rendered by default.

‎datafusion/core/src/physical_optimizer/mod.rs‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,13 @@ pub mod pipeline_checker;
3434
mod projection_pushdown;
3535
pub mod pruning;
3636
pub mod replace_with_order_preserving_variants;
37-
mod sort_pushdown;
37+
#[cfg(test)]
38+
pub mod test_utils;
3839
pub mod topk_aggregation;
39-
mod utils;
4040

4141
mod join_pipeline_selection;
42-
#[cfg(test)]
43-
pub mod test_utils;
42+
mod projection_pushdown;
43+
mod sort_pushdown;
44+
mod utils;
4445

4546
pub use optimizer::PhysicalOptimizerRule;

‎datafusion/core/src/physical_optimizer/pipeline_checker.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,14 @@ use std::sync::Arc;
2424
use crate::config::ConfigOptions;
2525
use crate::error::Result;
2626
use crate::physical_optimizer::PhysicalOptimizerRule;
27-
use crate::physical_plan::joins::SymmetricHashJoinExec;
2827
use crate::physical_plan::{with_new_children_if_necessary, ExecutionPlan};
2928

3029
use datafusion_common::config::OptimizerOptions;
3130
use datafusion_common::tree_node::{Transformed, TreeNode, VisitRecursion};
3231
use datafusion_common::{plan_err, DataFusionError};
3332
use datafusion_physical_expr::intervals::utils::{check_support, is_datatype_supported};
3433
use datafusion_physical_plan::joins::prunability::is_filter_expr_prunable;
34+
use datafusion_physical_plan::joins::SymmetricHashJoinExec;
3535

3636
/// The PipelineChecker rule rejects non-runnable query plans that use
3737
/// pipeline-breaking operators on infinite input(s).

‎datafusion/core/src/physical_optimizer/projection_pushdown.rs‎

Lines changed: 108 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
// Copyright (C) Synnada, Inc. - All Rights Reserved.
19+
// This file does not contain any Apache Software Foundation copyrighted code.
20+
1821
//! This file implements the `ProjectionPushdown` physical optimization rule.
1922
//! The function [`remove_unnecessary_projections`] tries to push down all
2023
//! projections one by one if the operator below is amenable to this. If a
@@ -29,10 +32,10 @@ use crate::datasource::physical_plan::CsvExec;
2932
use crate::error::Result;
3033
use crate::physical_plan::coalesce_partitions::CoalescePartitionsExec;
3134
use crate::physical_plan::filter::FilterExec;
32-
use crate::physical_plan::joins::utils::{ColumnIndex, JoinFilter};
35+
use crate::physical_plan::joins::utils::{ColumnIndex, JoinFilter, JoinSide};
3336
use crate::physical_plan::joins::{
34-
CrossJoinExec, HashJoinExec, NestedLoopJoinExec, SortMergeJoinExec,
35-
SymmetricHashJoinExec,
37+
CrossJoinExec, HashJoinExec, NestedLoopJoinExec, SlidingHashJoinExec,
38+
SortMergeJoinExec, SymmetricHashJoinExec,
3639
};
3740
use crate::physical_plan::memory::MemoryExec;
3841
use crate::physical_plan::projection::ProjectionExec;
@@ -133,6 +136,8 @@ pub fn remove_unnecessary_projections(
133136
try_swapping_with_cross_join(projection, cross_join)?
134137
} else if let Some(nl_join) = input.downcast_ref::<NestedLoopJoinExec>() {
135138
try_swapping_with_nested_loop_join(projection, nl_join)?
139+
} else if let Some(sh_join) = input.downcast_ref::<SlidingHashJoinExec>() {
140+
try_swapping_with_sliding_hash_join(projection, sh_join)?
136141
} else if let Some(sm_join) = input.downcast_ref::<SortMergeJoinExec>() {
137142
try_swapping_with_sort_merge_join(projection, sm_join)?
138143
} else if let Some(sym_join) = input.downcast_ref::<SymmetricHashJoinExec>() {
@@ -679,6 +684,85 @@ fn try_swapping_with_nested_loop_join(
679684
)?)))
680685
}
681686

687+
/// Tries to swap the projection with its input [`SlidingHashJoinExec`]. If it can be done,
688+
/// it returns the new swapped version having the [`SlidingHashJoinExec`] as the top plan.
689+
/// Otherwise, it returns None.
690+
fn try_swapping_with_sliding_hash_join(
691+
projection: &ProjectionExec,
692+
sh_join: &SlidingHashJoinExec,
693+
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
694+
// Convert projected PhysicalExpr's to columns. If not possible, we cannot proceed.
695+
let Some(projection_as_columns) = physical_to_column_exprs(projection.expr()) else {
696+
return Ok(None);
697+
};
698+
699+
let (far_right_left_col_ind, far_left_right_col_ind) = join_table_borders(
700+
sh_join.left().schema().fields().len(),
701+
&projection_as_columns,
702+
);
703+
704+
if !join_allows_pushdown(
705+
&projection_as_columns,
706+
sh_join.schema(),
707+
far_right_left_col_ind,
708+
far_left_right_col_ind,
709+
) {
710+
return Ok(None);
711+
}
712+
713+
let Some(new_on) = update_join_on(
714+
&projection_as_columns[0..=far_right_left_col_ind as _],
715+
&projection_as_columns[far_left_right_col_ind as _..],
716+
sh_join.on(),
717+
) else {
718+
return Ok(None);
719+
};
720+
721+
let Some(new_filter) = update_join_filter(
722+
&projection_as_columns[0..=far_right_left_col_ind as _],
723+
&projection_as_columns[far_left_right_col_ind as _..],
724+
sh_join.filter(),
725+
sh_join.left(),
726+
sh_join.right(),
727+
) else {
728+
return Ok(None);
729+
};
730+
731+
let (new_left_sort, new_right_sort) = match (
732+
update_sort_expr(
733+
sh_join.left_sort_exprs(),
734+
&projection.expr()[0..=far_right_left_col_ind as _],
735+
)?,
736+
update_sort_expr(
737+
sh_join.right_sort_exprs(),
738+
&projection.expr()[far_left_right_col_ind as _..],
739+
)?,
740+
) {
741+
(Some(left), Some(right)) => (left, right),
742+
_ => return Ok(None),
743+
};
744+
745+
let (new_left, new_right) = new_join_children(
746+
projection_as_columns,
747+
far_right_left_col_ind,
748+
far_left_right_col_ind,
749+
sh_join.left(),
750+
sh_join.right(),
751+
)?;
752+
753+
Ok(Some(Arc::new(SlidingHashJoinExec::try_new(
754+
Arc::new(new_left),
755+
Arc::new(new_right),
756+
new_on,
757+
new_filter,
758+
sh_join.join_type(),
759+
sh_join.null_equals_null(),
760+
new_left_sort,
761+
new_right_sort,
762+
*sh_join.partition_mode(),
763+
)?)))
764+
}
765+
682766
/// Tries to swap the projection with its input [`SortMergeJoinExec`]. If it can be done,
683767
/// it returns the new swapped version having the [`SortMergeJoinExec`] as the top plan.
684768
/// Otherwise, it returns None.
@@ -876,7 +960,6 @@ fn update_expr(
876960
/// references could not be.
877961
RewrittenInvalid,
878962
}
879-
880963
let mut state = RewriteState::Unchanged;
881964

882965
let new_expr = expr
@@ -1156,6 +1239,25 @@ fn new_join_children(
11561239
Ok((new_left, new_right))
11571240
}
11581241

1242+
/// [`PhysicalSortExpr`] handler version of update_expr() function.
1243+
fn update_sort_expr(
1244+
sort_exprs: &[PhysicalSortExpr],
1245+
projected_exprs: &[(Arc<dyn PhysicalExpr>, String)],
1246+
) -> Result<Option<Vec<PhysicalSortExpr>>> {
1247+
let mut new_sort_exprs = vec![];
1248+
for sort_expr in sort_exprs {
1249+
let Some(updated_expr) = update_expr(&sort_expr.expr, projected_exprs, false)?
1250+
else {
1251+
return Ok(None);
1252+
};
1253+
new_sort_exprs.push(PhysicalSortExpr {
1254+
expr: updated_expr,
1255+
options: sort_expr.options,
1256+
})
1257+
}
1258+
Ok(Some(new_sort_exprs))
1259+
}
1260+
11591261
#[cfg(test)]
11601262
mod tests {
11611263
use std::sync::Arc;
@@ -1170,7 +1272,7 @@ mod tests {
11701272
use crate::physical_optimizer::PhysicalOptimizerRule;
11711273
use crate::physical_plan::coalesce_partitions::CoalescePartitionsExec;
11721274
use crate::physical_plan::filter::FilterExec;
1173-
use crate::physical_plan::joins::utils::{ColumnIndex, JoinFilter};
1275+
use crate::physical_plan::joins::utils::{ColumnIndex, JoinFilter, JoinSide};
11741276
use crate::physical_plan::joins::StreamJoinPartitionMode;
11751277
use crate::physical_plan::memory::MemoryExec;
11761278
use crate::physical_plan::projection::ProjectionExec;
@@ -1192,7 +1294,7 @@ mod tests {
11921294
Distribution, Partitioning, PhysicalExpr, PhysicalSortExpr,
11931295
PhysicalSortRequirement, ScalarFunctionExpr,
11941296
};
1195-
use datafusion_physical_plan::joins::SymmetricHashJoinExec;
1297+
use datafusion_physical_plan::joins::{SymmetricHashJoinExec, SlidingHashJoinExec};
11961298
use datafusion_physical_plan::streaming::{PartitionStream, StreamingTableExec};
11971299
use datafusion_physical_plan::union::UnionExec;
11981300

‎datafusion/expr/src/signature.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ impl Signature {
249249
/// the function's behavior is monotonic, or non-monotonic/unknown for that argument, namely:
250250
/// - `None` signifies unknown monotonicity or non-monotonicity.
251251
/// - `Some(true)` indicates that the function is monotonically increasing w.r.t. the argument in question.
252-
/// - Some(false) indicates that the function is monotonically decreasing w.r.t. the argument in question.
252+
/// - `Some(false)` indicates that the function is monotonically decreasing w.r.t. the argument in question.
253253
pub type FuncMonotonicity = Vec<Option<bool>>;
254254

255255
#[cfg(test)]

‎datafusion/physical-expr/src/expressions/negative.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ use std::any::Any;
2121
use std::hash::{Hash, Hasher};
2222
use std::sync::Arc;
2323

24+
use crate::intervals::Interval;
2425
use crate::physical_expr::down_cast_any_ref;
2526
use crate::sort_properties::SortProperties;
2627
use crate::PhysicalExpr;

‎datafusion/physical-expr/src/functions.rs‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -30,26 +30,28 @@
3030
//! an argument i32 is passed to a function that supports f64, the
3131
//! argument is automatically is coerced to f64.
3232
33+
use std::ops::Neg;
34+
use std::sync::Arc;
35+
3336
use crate::execution_props::ExecutionProps;
37+
use crate::expressions::{cast_column, nullif_func};
3438
use crate::sort_properties::SortProperties;
3539
use crate::{
3640
array_expressions, conditional_expressions, datetime_expressions,
3741
expressions::nullif_func, math_expressions, string_expressions, struct_expressions,
3842
PhysicalExpr, ScalarFunctionExpr,
3943
};
44+
4045
use arrow::{
4146
array::ArrayRef,
4247
compute::kernels::length::{bit_length, length},
43-
datatypes::{DataType, Int32Type, Int64Type, Schema},
48+
datatypes::{DataType, Int32Type, Int64Type, Schema, TimeUnit},
4449
};
4550
use datafusion_common::{internal_err, DataFusionError, Result, ScalarValue};
46-
pub use datafusion_expr::FuncMonotonicity;
4751
use datafusion_expr::{
4852
type_coercion::functions::data_types, BuiltinScalarFunction, ColumnarValue,
4953
ScalarFunctionImplementation,
5054
};
51-
use std::ops::Neg;
52-
use std::sync::Arc;
5355

5456
/// Create a physical (function) expression.
5557
/// This function errors when `args`' can't be coerced to a valid argument type of the function.
@@ -979,8 +981,8 @@ fn func_order_in_one_dimension(
979981
#[cfg(test)]
980982
mod tests {
981983
use super::*;
982-
use crate::expressions::try_cast;
983-
use crate::expressions::{col, lit};
984+
use crate::expressions::{col, lit, try_cast};
985+
984986
use arrow::{
985987
array::{
986988
Array, ArrayRef, BinaryArray, BooleanArray, Float32Array, Float64Array,
@@ -990,8 +992,7 @@ mod tests {
990992
record_batch::RecordBatch,
991993
};
992994
use datafusion_common::cast::as_uint64_array;
993-
use datafusion_common::{exec_err, plan_err};
994-
use datafusion_common::{Result, ScalarValue};
995+
use datafusion_common::{exec_err, plan_err, Result, ScalarValue};
995996
use datafusion_expr::type_coercion::functions::data_types;
996997
use datafusion_expr::Signature;
997998

‎datafusion/sqllogictest/test_files/explain.slt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
255255
physical_plan CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], has_header=true
256256
physical_plan_with_stats CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], has_header=true, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]
257257

258-
259258
### tests for EXPLAIN with display statistics enabled
260259
statement ok
261260
set datafusion.explain.show_statistics = true;

‎datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt‎

Lines changed: 16 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -283,13 +283,12 @@ Sort: subquery.c_custkey ASC NULLS LAST, subquery.price_rank ASC NULLS LAST
283283
----------WindowAggr: windowExpr=[[ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW]]
284284
------------TableScan: customer projection=[c_custkey, c_nationkey]
285285
physical_plan
286-
ProjectionExec: expr=[c_custkey@2 as c_custkey, c_nationkey@3 as c_nationkey, price_rank@4 as price_rank, n_name@1 as n_name]
287-
--ProjectionExec: expr=[n_nationkey@3 as n_nationkey, n_name@4 as n_name, c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey, price_rank@2 as price_rank]
288-
----SortMergeJoin: join_type=Inner, on=[(c_custkey@0, n_nationkey@0)]
289-
------ProjectionExec: expr=[c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey, ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as price_rank]
290-
--------BoundedWindowAggExec: wdw=[ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Ok(Field { name: "ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", data_type: UInt64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), frame: WindowFrame { units: Range, start_bound: Preceding(Int64(NULL)), end_bound: CurrentRow }], mode=[Sorted]
291-
----------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/customer.csv]]}, projection=[c_custkey, c_nationkey], infinite_source=true, output_ordering=[c_custkey@0 ASC NULLS LAST], has_header=false
292-
------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/nation.csv]]}, projection=[n_nationkey, n_name], infinite_source=true, output_ordering=[n_nationkey@0 ASC NULLS LAST], has_header=false
286+
ProjectionExec: expr=[c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey, price_rank@2 as price_rank, n_name@4 as n_name]
287+
--SortMergeJoin: join_type=Inner, on=[(c_custkey@0, n_nationkey@0)]
288+
----ProjectionExec: expr=[c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey, ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as price_rank]
289+
------BoundedWindowAggExec: wdw=[ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Ok(Field { name: "ROW_NUMBER() ORDER BY [customer.c_custkey ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", data_type: UInt64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), frame: WindowFrame { units: Range, start_bound: Preceding(Int64(NULL)), end_bound: CurrentRow }], mode=[Sorted]
290+
--------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/customer.csv]]}, projection=[c_custkey, c_nationkey], infinite_source=true, output_ordering=[c_custkey@0 ASC NULLS LAST], has_header=false
291+
----CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/nation.csv]]}, projection=[n_nationkey, n_name], infinite_source=true, output_ordering=[n_nationkey@0 ASC NULLS LAST], has_header=false
293292

294293
query TT
295294
EXPLAIN SELECT
@@ -374,18 +373,16 @@ Sort: nation.n_nationkey ASC NULLS LAST
374373
----TableScan: nation projection=[n_nationkey, n_name, n_regionkey, n_comment, n_rev]
375374
physical_plan
376375
SlidingNestedLoopJoinExec: join_type=Inner, filter=n_nationkey@1 > c_custkey@0 AND n_nationkey@1 < c_custkey@0 + 20
377-
--ProjectionExec: expr=[c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey]
378-
----ProjectionExec: expr=[c_custkey@0 as c_custkey, c_nationkey@1 as c_nationkey, o_orderkey@2 as o_orderkey]
379-
------ProjectionExec: expr=[c_custkey@2 as c_custkey, c_nationkey@3 as c_nationkey, o_orderkey@4 as o_orderkey, o_orderdate@5 as o_orderdate, l_orderkey@0 as l_orderkey, l_shipdate@1 as l_shipdate]
380-
--------SlidingHashJoinExec: join_type=Inner, on=[(l_shipdate@1, o_orderdate@3)], filter=l_orderkey@1 >= o_orderkey@0 - 10 AND l_orderkey@1 < o_orderkey@0 + 10
381-
----------ProjectionExec: expr=[l_orderkey@0 as l_orderkey, l_shipdate@2 as l_shipdate]
382-
------------CoalesceBatchesExec: target_batch_size=8192
383-
--------------FilterExec: l_returnflag@1 = R
384-
----------------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/lineitem.csv]]}, projection=[l_orderkey, l_returnflag, l_shipdate], infinite_source=true, output_ordering=[l_orderkey@0 ASC NULLS LAST], has_header=false
385-
----------ProjectionExec: expr=[c_custkey@2 as c_custkey, c_nationkey@3 as c_nationkey, o_orderkey@0 as o_orderkey, o_orderdate@1 as o_orderdate]
386-
------------SortMergeJoin: join_type=Inner, on=[(o_orderkey@0, c_custkey@0)]
387-
--------------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/orders.csv]]}, projection=[o_orderkey, o_orderdate], infinite_source=true, output_ordering=[o_orderkey@0 ASC NULLS LAST], has_header=false
388-
--------------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/customer.csv]]}, projection=[c_custkey, c_nationkey], infinite_source=true, output_ordering=[c_custkey@0 ASC NULLS LAST], has_header=false
376+
--ProjectionExec: expr=[c_custkey@2 as c_custkey, c_nationkey@3 as c_nationkey]
377+
----SlidingHashJoinExec: join_type=Inner, on=[(l_shipdate@1, o_orderdate@3)], filter=l_orderkey@1 >= o_orderkey@0 - 10 AND l_orderkey@1 < o_orderkey@0 + 10
378+
------ProjectionExec: expr=[l_orderkey@0 as l_orderkey, l_shipdate@2 as l_shipdate]
379+
--------CoalesceBatchesExec: target_batch_size=8192
380+
----------FilterExec: l_returnflag@1 = R
381+
------------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/lineitem.csv]]}, projection=[l_orderkey, l_returnflag, l_shipdate], infinite_source=true, output_ordering=[l_orderkey@0 ASC NULLS LAST], has_header=false
382+
------ProjectionExec: expr=[c_custkey@2 as c_custkey, c_nationkey@3 as c_nationkey, o_orderkey@0 as o_orderkey, o_orderdate@1 as o_orderdate]
383+
--------SortMergeJoin: join_type=Inner, on=[(o_orderkey@0, c_custkey@0)]
384+
----------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/orders.csv]]}, projection=[o_orderkey, o_orderdate], infinite_source=true, output_ordering=[o_orderkey@0 ASC NULLS LAST], has_header=false
385+
----------CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/customer.csv]]}, projection=[c_custkey, c_nationkey], infinite_source=true, output_ordering=[c_custkey@0 ASC NULLS LAST], has_header=false
389386
--CsvExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/tpch-csv/nation.csv]]}, projection=[n_nationkey, n_name, n_regionkey, n_comment, n_rev], infinite_source=true, output_ordering=[n_nationkey@0 ASC NULLS LAST], has_header=false
390387

391388
query TT

0 commit comments

Comments
 (0)