Repository navigation
Commit ca1561f
perf: avoid building a DFSchema for every pruning comparison (apache#26405)
## Which issue does this PR close?
- Related to apache#21968. There is no dedicated issue for this change.
## Rationale for this change
Building a pruning predicate for a Parquet file with a wide schema takes
time proportional to the number of columns in the file, once for every
comparison in the predicate, even when the predicate only references one
column. A Parquet scan builds several pruning predicates for each file
it opens (for example for row group statistics, the page index and bloom
filters), so on wide tables this cost is paid many times per file.
The cause is in `PruningExpressionBuilder::try_new`. For every
comparison it converts the whole file schema into a `DFSchema`, which
runs `DFSchema::check_names` and inserts every field name into a
`BTreeSet`. The `DFSchema` is then only used through
`schema.as_arrow()`, to get the data type of the expression inside a
`CAST` or `TRY_CAST`. That is the Arrow schema the builder already has.
In a profile of `wide_schema` Q01 (1024 columns, 256 files),
`DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was
about 5% of CPU samples. With a file metadata cache large enough to hold
every footer it was 9.8%.
The `DFSchema` dates from apache#3422, when pruning rewrote logical `Expr`s
and needed a `DFSchema` to resolve their types. apache#5419 moved pruning to
physical expressions, which only need the Arrow schema, but kept the
`DFSchema` parameter.
## What changes are included in this PR?
- `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take
`&Schema` instead of a `DFSchema`.
- `PruningExpressionBuilder::try_new` passes the schema it already has,
rather than building a `DFSchema` for each comparison.
- Two unit tests that built a `DFSchema` only to call
`rewrite_expr_to_prunable` now pass the `Schema`.
## What is the testing strategy for this PR?
The change is covered by existing tests, so this PR adds none. Both
functions are private, and their behaviour is unchanged apart from the
duplicate-name case described below.
- `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable`
and `test_rewrite_expr_to_prunable_error`
- `datafusion-datasource-parquet` lib tests
- `datafusion` `parquet_integration` tests, which cover row group and
page index pruning
## Are there any user-facing changes?
There are no API changes, because both functions are private.
There is one behaviour change. `DFSchema::try_from` fails when the
schema has duplicate field names, so previously no comparison against
such a schema could be used for pruning. Now pruning goes ahead and
resolves the column by name to the first field with that name. The
column lookup in the same function (`schema.column_with_name`) and the
Parquet statistics lookup (`parquet_column`) already resolve names this
way.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 37c954c commit ca1561f
1 file changed
Lines changed: 16 additions & 40 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
45 | | - | |
46 | | - | |
| 44 | + | |
47 | 45 | | |
48 | 46 | | |
49 | 47 | | |
| |||
1155 | 1153 | | |
1156 | 1154 | | |
1157 | 1155 | | |
1158 | | - | |
1159 | | - | |
1160 | | - | |
1161 | | - | |
1162 | | - | |
1163 | | - | |
1164 | | - | |
| 1156 | + | |
| 1157 | + | |
1165 | 1158 | | |
1166 | 1159 | | |
1167 | 1160 | | |
| |||
1251 | 1244 | | |
1252 | 1245 | | |
1253 | 1246 | | |
1254 | | - | |
| 1247 | + | |
1255 | 1248 | | |
1256 | 1249 | | |
1257 | 1250 | | |
| |||
1329 | 1322 | | |
1330 | 1323 | | |
1331 | 1324 | | |
1332 | | - | |
| 1325 | + | |
1333 | 1326 | | |
1334 | | - | |
1335 | | - | |
1336 | | - | |
1337 | | - | |
| 1327 | + | |
1338 | 1328 | | |
1339 | 1329 | | |
1340 | 1330 | | |
| |||
6753 | 6743 | | |
6754 | 6744 | | |
6755 | 6745 | | |
6756 | | - | |
6757 | 6746 | | |
6758 | 6747 | | |
6759 | 6748 | | |
6760 | 6749 | | |
6761 | 6750 | | |
6762 | 6751 | | |
6763 | | - | |
6764 | | - | |
6765 | | - | |
6766 | | - | |
6767 | | - | |
6768 | | - | |
6769 | | - | |
| 6752 | + | |
| 6753 | + | |
| 6754 | + | |
6770 | 6755 | | |
6771 | 6756 | | |
6772 | 6757 | | |
| |||
6775 | 6760 | | |
6776 | 6761 | | |
6777 | 6762 | | |
6778 | | - | |
6779 | | - | |
6780 | | - | |
6781 | | - | |
6782 | | - | |
6783 | | - | |
6784 | | - | |
| 6763 | + | |
| 6764 | + | |
| 6765 | + | |
6785 | 6766 | | |
6786 | 6767 | | |
6787 | 6768 | | |
| |||
6791 | 6772 | | |
6792 | 6773 | | |
6793 | 6774 | | |
6794 | | - | |
| 6775 | + | |
6795 | 6776 | | |
6796 | 6777 | | |
6797 | 6778 | | |
| |||
6872 | 6853 | | |
6873 | 6854 | | |
6874 | 6855 | | |
6875 | | - | |
6876 | 6856 | | |
6877 | 6857 | | |
6878 | 6858 | | |
6879 | 6859 | | |
6880 | | - | |
6881 | | - | |
6882 | | - | |
6883 | | - | |
6884 | | - | |
6885 | | - | |
| 6860 | + | |
| 6861 | + | |
6886 | 6862 | | |
6887 | 6863 | | |
6888 | 6864 | | |
| |||
6891 | 6867 | | |
6892 | 6868 | | |
6893 | 6869 | | |
6894 | | - | |
| 6870 | + | |
6895 | 6871 | | |
6896 | 6872 | | |
6897 | 6873 | | |
| |||
0 commit comments