Skip to content

Commit ca1561f

Browse files
cetra3claude
andauthored
perf: avoid building a DFSchema for every pruning comparison (apache#26405)
## Which issue does this PR close? - Related to apache#21968. There is no dedicated issue for this change. ## Rationale for this change Building a pruning predicate for a Parquet file with a wide schema takes time proportional to the number of columns in the file, once for every comparison in the predicate, even when the predicate only references one column. A Parquet scan builds several pruning predicates for each file it opens (for example for row group statistics, the page index and bloom filters), so on wide tables this cost is paid many times per file. The cause is in `PruningExpressionBuilder::try_new`. For every comparison it converts the whole file schema into a `DFSchema`, which runs `DFSchema::check_names` and inserts every field name into a `BTreeSet`. The `DFSchema` is then only used through `schema.as_arrow()`, to get the data type of the expression inside a `CAST` or `TRY_CAST`. That is the Arrow schema the builder already has. In a profile of `wide_schema` Q01 (1024 columns, 256 files), `DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was about 5% of CPU samples. With a file metadata cache large enough to hold every footer it was 9.8%. The `DFSchema` dates from apache#3422, when pruning rewrote logical `Expr`s and needed a `DFSchema` to resolve their types. apache#5419 moved pruning to physical expressions, which only need the Arrow schema, but kept the `DFSchema` parameter. ## What changes are included in this PR? - `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take `&Schema` instead of a `DFSchema`. - `PruningExpressionBuilder::try_new` passes the schema it already has, rather than building a `DFSchema` for each comparison. - Two unit tests that built a `DFSchema` only to call `rewrite_expr_to_prunable` now pass the `Schema`. ## What is the testing strategy for this PR? The change is covered by existing tests, so this PR adds none. Both functions are private, and their behaviour is unchanged apart from the duplicate-name case described below. - `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable` and `test_rewrite_expr_to_prunable_error` - `datafusion-datasource-parquet` lib tests - `datafusion` `parquet_integration` tests, which cover row group and page index pruning ## Are there any user-facing changes? There are no API changes, because both functions are private. There is one behaviour change. `DFSchema::try_from` fails when the schema has duplicate field names, so previously no comparison against such a schema could be used for pruning. Now pruning goes ahead and resolves the column by name to the first field with that name. The column lookup in the same function (`schema.column_with_name`) and the Parquet statistics lookup (`parquet_column`) already resolve names this way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 37c954c commit ca1561f

1 file changed

Lines changed: 16 additions & 40 deletions

File tree

‎datafusion/pruning/src/pruning_predicate.rs‎

Lines changed: 16 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,7 @@ use log::{debug, trace};
4141

4242
use datafusion_common::error::Result;
4343
use datafusion_common::tree_node::{TransformedResult, TreeNodeRecursion};
44-
use datafusion_common::{
45-
_internal_datafusion_err, Column, DFSchema, assert_eq_or_internal_err,
46-
};
44+
use datafusion_common::{_internal_datafusion_err, Column, assert_eq_or_internal_err};
4745
use datafusion_common::{
4846
ScalarValue, internal_datafusion_err, plan_datafusion_err, plan_err,
4947
tree_node::{Transformed, TreeNode},
@@ -1155,13 +1153,8 @@ impl<'a> PruningExpressionBuilder<'a> {
11551153
}
11561154
};
11571155

1158-
let df_schema = DFSchema::try_from(Arc::clone(schema))?;
1159-
let (column_expr, correct_operator, scalar_expr) = rewrite_expr_to_prunable(
1160-
column_expr,
1161-
correct_operator,
1162-
scalar_expr,
1163-
df_schema,
1164-
)?;
1156+
let (column_expr, correct_operator, scalar_expr) =
1157+
rewrite_expr_to_prunable(column_expr, correct_operator, scalar_expr, schema)?;
11651158
let Some((_, field)) = schema.column_with_name(column.name()) else {
11661159
return plan_err!("Field not found in schema");
11671160
};
@@ -1251,7 +1244,7 @@ fn rewrite_expr_to_prunable(
12511244
column_expr: &PhysicalExprRef,
12521245
op: Operator,
12531246
scalar_expr: &PhysicalExprRef,
1254-
schema: DFSchema,
1247+
schema: &Schema,
12551248
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
12561249
if !is_compare_op(op) {
12571250
return plan_err!("rewrite_expr_to_prunable only support compare expression");
@@ -1329,12 +1322,9 @@ fn rewrite_cast_child_to_prunable(
13291322
cast_type: &DataType,
13301323
op: Operator,
13311324
scalar_expr: &PhysicalExprRef,
1332-
schema: DFSchema,
1325+
schema: &Schema,
13331326
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
1334-
verify_support_type_for_prune(
1335-
&cast_child_expr.data_type(schema.as_arrow())?,
1336-
cast_type,
1337-
)?;
1327+
verify_support_type_for_prune(&cast_child_expr.data_type(schema)?, cast_type)?;
13381328
rewrite_expr_to_prunable(cast_child_expr, op, scalar_expr, schema)
13391329
}
13401330

@@ -6753,20 +6743,15 @@ mod tests {
67536743
#[test]
67546744
fn test_rewrite_expr_to_prunable() {
67556745
let schema = Schema::new(vec![Field::new("a", DataType::Int32, true)]);
6756-
let df_schema = DFSchema::try_from(schema.clone()).unwrap();
67576746

67586747
// column op lit
67596748
let left_input = col("a");
67606749
let left_input = logical2physical(&left_input, &schema);
67616750
let right_input = lit(ScalarValue::Int32(Some(12)));
67626751
let right_input = logical2physical(&right_input, &schema);
6763-
let (result_left, _, result_right) = rewrite_expr_to_prunable(
6764-
&left_input,
6765-
Operator::Eq,
6766-
&right_input,
6767-
df_schema.clone(),
6768-
)
6769-
.unwrap();
6752+
let (result_left, _, result_right) =
6753+
rewrite_expr_to_prunable(&left_input, Operator::Eq, &right_input, &schema)
6754+
.unwrap();
67706755
assert_eq!(result_left.to_string(), left_input.to_string());
67716756
assert_eq!(result_right.to_string(), right_input.to_string());
67726757

@@ -6775,13 +6760,9 @@ mod tests {
67756760
let left_input = logical2physical(&left_input, &schema);
67766761
let right_input = lit(ScalarValue::Decimal128(Some(12), 20, 3));
67776762
let right_input = logical2physical(&right_input, &schema);
6778-
let (result_left, _, result_right) = rewrite_expr_to_prunable(
6779-
&left_input,
6780-
Operator::Gt,
6781-
&right_input,
6782-
df_schema.clone(),
6783-
)
6784-
.unwrap();
6763+
let (result_left, _, result_right) =
6764+
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema)
6765+
.unwrap();
67856766
assert_eq!(result_left.to_string(), left_input.to_string());
67866767
assert_eq!(result_right.to_string(), right_input.to_string());
67876768

@@ -6791,7 +6772,7 @@ mod tests {
67916772
let right_input = lit(ScalarValue::Int64(Some(12)));
67926773
let right_input = logical2physical(&right_input, &schema);
67936774
let (result_left, _, result_right) =
6794-
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, df_schema)
6775+
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema)
67956776
.unwrap();
67966777
assert_eq!(result_left.to_string(), left_input.to_string());
67976778
assert_eq!(result_right.to_string(), right_input.to_string());
@@ -6872,17 +6853,12 @@ mod tests {
68726853
// cast string value to numeric value
68736854
// this cast is not supported
68746855
let schema = Schema::new(vec![Field::new("a", DataType::Utf8, true)]);
6875-
let df_schema = DFSchema::try_from(schema.clone()).unwrap();
68766856
let left_input = cast(col("a"), DataType::Int64);
68776857
let left_input = logical2physical(&left_input, &schema);
68786858
let right_input = lit(ScalarValue::Int64(Some(12)));
68796859
let right_input = logical2physical(&right_input, &schema);
6880-
let result = rewrite_expr_to_prunable(
6881-
&left_input,
6882-
Operator::Gt,
6883-
&right_input,
6884-
df_schema.clone(),
6885-
);
6860+
let result =
6861+
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema);
68866862
assert!(result.is_err());
68876863

68886864
// other expr
@@ -6891,7 +6867,7 @@ mod tests {
68916867
let right_input = lit(ScalarValue::Int64(Some(12)));
68926868
let right_input = logical2physical(&right_input, &schema);
68936869
let result =
6894-
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, df_schema);
6870+
rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema);
68956871
assert!(result.is_err());
68966872
// TODO: add other negative test for other case and op
68976873
}

0 commit comments

Comments
 (0)