Repository navigation
support cast/try_cast in prune with signed integer and decimal - #3422
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3422 +/- ##
==========================================
+ Coverage 85.58% 85.66% +0.07%
==========================================
Files 296 296
Lines 54252 54544 +292
==========================================
+ Hits 46432 46723 +291
- Misses 7820 7821 +1
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| ], | ||
| negated: true, | ||
| }; | ||
| let expected_expr = "CAST(#c1_min AS Int64) != Int64(1) OR Int64(1) != CAST(#c1_max AS Int64) AND CAST(#c1_min AS Int64) != Int64(2) OR Int64(2) != CAST(#c1_max AS Int64) AND CAST(#c1_min AS Int64) != Int64(3) OR Int64(3) != CAST(#c1_max AS Int64)"; |
There was a problem hiding this comment.
I'm not sure I understand what is happening with the negated case. Could you add a comment here explaining this? Why is there != in here rather than <= or >=?
There was a problem hiding this comment.
@andygrove I don't change any logical about the list.
you can get this from #2282 (comment)
I get the meanings from the code context.
if the min_c1 and max_c1 is 1, all the value in the column or chunk must be 1 or null, so the c1 not in (1) can skip this column or chunk.
alamb
left a comment
There was a problem hiding this comment.
Thank you @liukun4515 -- this is great
I will look at this code in the morning as well, but I think we need to handle the case where casting won't preserve sortedness (aka UTF8 -> numeric), as described on #3377
I got your point. We can add rule for cast/try_cast case. For example, just support cast/try_cast from numeric to numeric data type, I think the data type of numeric has the order. We can support other complex case in the future pr. |
|
cc @alamb |
| if support_type_for_prune(&from_type, data_type) { | ||
| let (left, op, right) = | ||
| rewrite_expr_to_prunable(expr, op, scalar_expr, schema)?; | ||
| Ok((cast(left, data_type.clone()), op, right)) | ||
| } else { | ||
| return Err(DataFusionError::Plan(format!( | ||
| "Cast with from type {} to type {} is not supported", | ||
| from_type, data_type | ||
| ))); | ||
| } |
There was a problem hiding this comment.
We could probably avoid the duplication of the error generation by moving the error return into support_type_for_prune like:
| if support_type_for_prune(&from_type, data_type) { | |
| let (left, op, right) = | |
| rewrite_expr_to_prunable(expr, op, scalar_expr, schema)?; | |
| Ok((cast(left, data_type.clone()), op, right)) | |
| } else { | |
| return Err(DataFusionError::Plan(format!( | |
| "Cast with from type {} to type {} is not supported", | |
| from_type, data_type | |
| ))); | |
| } | |
| verify_support_type_for_prune(&from_type, data_type)?; | |
| let (left, op, right) = rewrite_expr_to_prunable(expr, op, scalar_expr)?; | |
| Ok((cast(left, data_type.clone()), op, right)) |
This is not needed for correctness, just a code structure suggestion
|
|
||
| // The pruning logic is based on the comparing the min/max bounds. | ||
| // Must make sure the two type has order. | ||
| // For example, casts from string to numbers is not correct. |
There was a problem hiding this comment.
👍 thank you for these comments
| // For example, casts from string to numbers is not correct. | ||
| // Because the "13" is less than "3" with UTF8 comparison order. | ||
| fn support_type_for_prune(from_type: &DataType, to_type: &DataType) -> bool { | ||
| // TODO: support other data type |
There was a problem hiding this comment.
I think it would be good to file a follow on ticket listing the other types (you did so in #3414 (comment))
| rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, df_schema); | ||
| assert!(result.is_err()); | ||
| } | ||
| } |
There was a problem hiding this comment.
I think we should add some tests that exercise the pruning code with actual values too
For example, perhaps you can take prune_int32_cast from #3378?
| #[test] | ||
| fn test_rewrite_expr_to_prunable_error() { | ||
| // cast string value to numeric value | ||
| // this cast is not supported |
| .unwrap(); | ||
| assert_eq!(result_left, left_input); | ||
| assert_eq!(result_right, right_input); | ||
| // TODO: add test for other case and op |
There was a problem hiding this comment.
Rather than extending this test for other ops, I think we should add tests which actually run the pruning expressions on values (see my other comments).
Would it be helpful for me to write some more tests? I can do so tomorrow morning
There was a problem hiding this comment.
I find i have added the test for executing with real data in the prune_cast_column_scalar function.
You can check them, it just include cast(col, int64) and try_cast(col,int64). If you think it is not enough, you can add more test for it or give some advices for the test.
|
To be explicit I think the code in this PR looks good to me, but I would like to see some actual tests with pruning expressions on data prior to merging it. I am happy to help write these tests too |
@alamb Thanks for you help. |
4b1a405 to
eb9a540
Compare
| rewrite_expr_to_prunable(expr, op, scalar_expr, schema)?; | ||
| Ok((try_cast(left, data_type.clone()), op, right)) | ||
| } | ||
| // `col > lit()` |
There was a problem hiding this comment.
This comment should be removed I think?
eb9a540 to
706357e
Compare
alamb
left a comment
There was a problem hiding this comment.
I re-reviewed this @liukun4515 and I think it looks good to go. Thank you.
I wrote several more tests which I will put up in a follow on PR but they all pass and I don't think there is any reason to hold this PR.
|
Follow on PR is #3454 |
|
Benchmark runs are scheduled for baseline = 3b8a20a and contender = 69d05aa. 69d05aa is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
Thanks for your careful review. @alamb |
…26405) ## Which issue does this PR close? - Related to apache#21968. There is no dedicated issue for this change. ## Rationale for this change Building a pruning predicate for a Parquet file with a wide schema takes time proportional to the number of columns in the file, once for every comparison in the predicate, even when the predicate only references one column. A Parquet scan builds several pruning predicates for each file it opens (for example for row group statistics, the page index and bloom filters), so on wide tables this cost is paid many times per file. The cause is in `PruningExpressionBuilder::try_new`. For every comparison it converts the whole file schema into a `DFSchema`, which runs `DFSchema::check_names` and inserts every field name into a `BTreeSet`. The `DFSchema` is then only used through `schema.as_arrow()`, to get the data type of the expression inside a `CAST` or `TRY_CAST`. That is the Arrow schema the builder already has. In a profile of `wide_schema` Q01 (1024 columns, 256 files), `DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was about 5% of CPU samples. With a file metadata cache large enough to hold every footer it was 9.8%. The `DFSchema` dates from apache#3422, when pruning rewrote logical `Expr`s and needed a `DFSchema` to resolve their types. apache#5419 moved pruning to physical expressions, which only need the Arrow schema, but kept the `DFSchema` parameter. ## What changes are included in this PR? - `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take `&Schema` instead of a `DFSchema`. - `PruningExpressionBuilder::try_new` passes the schema it already has, rather than building a `DFSchema` for each comparison. - Two unit tests that built a `DFSchema` only to call `rewrite_expr_to_prunable` now pass the `Schema`. ## What is the testing strategy for this PR? The change is covered by existing tests, so this PR adds none. Both functions are private, and their behaviour is unchanged apart from the duplicate-name case described below. - `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable` and `test_rewrite_expr_to_prunable_error` - `datafusion-datasource-parquet` lib tests - `datafusion` `parquet_integration` tests, which cover row group and page index pruning ## Are there any user-facing changes? There are no API changes, because both functions are private. There is one behaviour change. `DFSchema::try_from` fails when the schema has duplicate field names, so previously no comparison against such a schema could be used for pruning. Now pruning goes ahead and resolves the column by name to the first field with that name. The column lookup in the same function (`schema.column_with_name`) and the Parquet statistics lookup (`parquet_column`) already resolve names this way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
…26405) ## Which issue does this PR close? - Related to apache#21968. There is no dedicated issue for this change. ## Rationale for this change Building a pruning predicate for a Parquet file with a wide schema takes time proportional to the number of columns in the file, once for every comparison in the predicate, even when the predicate only references one column. A Parquet scan builds several pruning predicates for each file it opens (for example for row group statistics, the page index and bloom filters), so on wide tables this cost is paid many times per file. The cause is in `PruningExpressionBuilder::try_new`. For every comparison it converts the whole file schema into a `DFSchema`, which runs `DFSchema::check_names` and inserts every field name into a `BTreeSet`. The `DFSchema` is then only used through `schema.as_arrow()`, to get the data type of the expression inside a `CAST` or `TRY_CAST`. That is the Arrow schema the builder already has. In a profile of `wide_schema` Q01 (1024 columns, 256 files), `DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was about 5% of CPU samples. With a file metadata cache large enough to hold every footer it was 9.8%. The `DFSchema` dates from apache#3422, when pruning rewrote logical `Expr`s and needed a `DFSchema` to resolve their types. apache#5419 moved pruning to physical expressions, which only need the Arrow schema, but kept the `DFSchema` parameter. ## What changes are included in this PR? - `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take `&Schema` instead of a `DFSchema`. - `PruningExpressionBuilder::try_new` passes the schema it already has, rather than building a `DFSchema` for each comparison. - Two unit tests that built a `DFSchema` only to call `rewrite_expr_to_prunable` now pass the `Schema`. ## What is the testing strategy for this PR? The change is covered by existing tests, so this PR adds none. Both functions are private, and their behaviour is unchanged apart from the duplicate-name case described below. - `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable` and `test_rewrite_expr_to_prunable_error` - `datafusion-datasource-parquet` lib tests - `datafusion` `parquet_integration` tests, which cover row group and page index pruning ## Are there any user-facing changes? There are no API changes, because both functions are private. There is one behaviour change. `DFSchema::try_from` fails when the schema has duplicate field names, so previously no comparison against such a schema could be used for pruning. Now pruning goes ahead and resolves the column by name to the first field with that name. The column lookup in the same function (`schema.column_with_name`) and the Parquet statistics lookup (`parquet_column`) already resolve names this way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
Which issue does this PR close?
part of #3414
Closes #3377
In order to merge this #3396, we should support the cast/try_cast in the prune first.
cc @andygrove @alamb
Just support cast from unsigned integer or decimal to unsigned integer or decimal.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?