Repository navigation
Commit 641e6cc
perf(parquet): build row-filter candidates once per file, share to stealers
`build_row_filter` does two things at very different costs:
1. Walk the predicate, split conjuncts, build a `FilterCandidate` per
conjunct (resolves ProjectionMask + projected schema + required-bytes
estimate), and optionally reorder by cost. This is schema/metadata
work that is identical for every open of the same file.
2. Bind each candidate to the current open's metrics counters.
Before this change, both ran per open — so a 226-RG file split into
226 chunks paid the analysis cost 226×. After this change, the donor
(or an un-split file open) builds the `Vec<FilterCandidate>` once in
`prepare_filters`; donated chunks carry it through `ParquetOpenChunk`;
each `build_stream` does only the cheap metric binding via the new
`row_filter_from_candidates`.
Refactor:
- Split `row_filter::build_row_filter` into
`build_row_filter_candidates` (expensive, metrics-free) and
`row_filter_from_candidates` (cheap, per-open). `build_row_filter`
becomes a thin wrapper.
- `FilterCandidate` now `Clone`.
- `FiltersPreparedParquetOpen` gains
`row_filter_candidates: Option<Arc<Vec<FilterCandidate>>>`, built in
`prepare_filters` from the donor's rewritten predicate.
- `ParquetOpenChunk` carries the same `Arc` across the handoff so
stealers reuse it in `build_stream`.
- `build_stream` now calls `row_filter_from_candidates` on the cached
vec instead of re-running the full builder.
Correctness: each open still gets its own metric bindings — only the
candidate analysis is shared. Existing tests pass (103 lib +
200 integration).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 16967cd commit 641e6cc
2 files changed
Lines changed: 91 additions & 31 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
25 | | - | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
| |||
326 | 329 | | |
327 | 330 | | |
328 | 331 | | |
| 332 | + | |
329 | 333 | | |
330 | 334 | | |
331 | 335 | | |
| |||
353 | 357 | | |
354 | 358 | | |
355 | 359 | | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
356 | 365 | | |
357 | 366 | | |
358 | 367 | | |
| |||
769 | 778 | | |
770 | 779 | | |
771 | 780 | | |
| 781 | + | |
772 | 782 | | |
773 | 783 | | |
774 | 784 | | |
| |||
816 | 826 | | |
817 | 827 | | |
818 | 828 | | |
| 829 | + | |
819 | 830 | | |
820 | 831 | | |
821 | 832 | | |
| |||
999 | 1010 | | |
1000 | 1011 | | |
1001 | 1012 | | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
1002 | 1039 | | |
1003 | 1040 | | |
1004 | 1041 | | |
| |||
1007 | 1044 | | |
1008 | 1045 | | |
1009 | 1046 | | |
| 1047 | + | |
1010 | 1048 | | |
1011 | 1049 | | |
1012 | 1050 | | |
| |||
1282 | 1320 | | |
1283 | 1321 | | |
1284 | 1322 | | |
| 1323 | + | |
1285 | 1324 | | |
1286 | 1325 | | |
1287 | 1326 | | |
| |||
1309 | 1348 | | |
1310 | 1349 | | |
1311 | 1350 | | |
| 1351 | + | |
1312 | 1352 | | |
1313 | 1353 | | |
1314 | 1354 | | |
| |||
1319 | 1359 | | |
1320 | 1360 | | |
1321 | 1361 | | |
1322 | | - | |
1323 | | - | |
1324 | | - | |
1325 | | - | |
1326 | | - | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
1327 | 1367 | | |
1328 | | - | |
1329 | | - | |
1330 | | - | |
1331 | | - | |
1332 | | - | |
1333 | | - | |
1334 | | - | |
1335 | | - | |
1336 | | - | |
1337 | | - | |
1338 | | - | |
| 1368 | + | |
| 1369 | + | |
1339 | 1370 | | |
1340 | | - | |
| 1371 | + | |
1341 | 1372 | | |
1342 | 1373 | | |
1343 | 1374 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| 180 | + | |
180 | 181 | | |
181 | 182 | | |
182 | 183 | | |
| |||
1019 | 1020 | | |
1020 | 1021 | | |
1021 | 1022 | | |
1022 | | - | |
1023 | | - | |
1024 | | - | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
1025 | 1030 | | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
1026 | 1045 | | |
1027 | 1046 | | |
1028 | 1047 | | |
| |||
1039 | 1058 | | |
1040 | 1059 | | |
1041 | 1060 | | |
1042 | | - | |
1043 | 1061 | | |
1044 | 1062 | | |
1045 | 1063 | | |
1046 | 1064 | | |
1047 | 1065 | | |
1048 | 1066 | | |
1049 | 1067 | | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
1050 | 1084 | | |
1051 | 1085 | | |
1052 | 1086 | | |
| |||
1055 | 1089 | | |
1056 | 1090 | | |
1057 | 1091 | | |
1058 | | - | |
| 1092 | + | |
1059 | 1093 | | |
1060 | 1094 | | |
1061 | 1095 | | |
1062 | | - | |
1063 | | - | |
1064 | 1096 | | |
1065 | | - | |
1066 | | - | |
1067 | 1097 | | |
1068 | 1098 | | |
1069 | 1099 | | |
1070 | 1100 | | |
1071 | 1101 | | |
1072 | | - | |
1073 | 1102 | | |
1074 | | - | |
| 1103 | + | |
1075 | 1104 | | |
1076 | 1105 | | |
1077 | 1106 | | |
1078 | 1107 | | |
1079 | 1108 | | |
1080 | 1109 | | |
1081 | 1110 | | |
1082 | | - | |
| 1111 | + | |
1083 | 1112 | | |
1084 | 1113 | | |
1085 | 1114 | | |
| |||
0 commit comments