Repository navigation
Prove comparisons true for every row from statistics - #10686
robert3005 wants to merge 1 commit into
Conversation
Merging this PR will degrade performance by 16.9%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
| let rhs = expr.child(1); | ||
|
|
||
| let value_predicate = match operator { | ||
| Operator::Gt => min(lhs).zip(max(rhs)).map(|(a, b)| gt(a, b)), |
There was a problem hiding this comment.
This performs some non-trivial work, let's add a cache.
|
Let's merge this after df dynamic expression |
Stats rewrites could only falsify a predicate. A scan can also skip a predicate that statistics prove true for every row, for example a hash join bound that covers the whole range of a column in a file. - Binary comparisons, AND, OR and BETWEEN get `satisfy` rules. A comparison is proven when the bounds of its operands order every pair of values and neither operand holds a null or a NaN. - `VortexFile::is_satisfied` proves a filter from the file statistics. - `VortexFile::is_satisfied_by_non_null_rows` proves it for the rows in which no column is null. This holds when a filter would only remove rows with nulls. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Robert Kruszewski <[email protected]>
58d5cc5 to
e5d3cc8
Compare
…11114 Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with. Signed-off-by: Connor Tsui <[email protected]>
Before this change, stats rewrites could only falsify a predicate. A scan can also skip a predicate that statistics prove true for every row. An example is a hash join bound that covers the full range of a column in a file.
satisfyrules. A comparison is proven true when the operand bounds order every pair of values, and neither operand holds a null or a NaN.VortexFile::is_satisfiedproves a filter from the file statistics.VortexFile::is_satisfied_by_non_null_rowsproves a filter for the rows in which no column is null. It binds every null count to zero. The min and max statistics ignore nulls, so the proof is correct for those rows.