Skip to content

Commit 2a42140

Browse files
adriangbclaude
andauthored
bench(predicate_eval): add expensive-first and wide-column shapes (apache#25032)
## Which issue does this PR close? - Part of apache#22883 - Related to apache#11262 - Prequel split out of apache#22698: these are the benchmark-suite additions from that PR, carried on their own so they can land (and be run) independently of the adaptive-filter work. ## Rationale for this change The `predicate_eval` suite (the conjunctive-filter microbenchmarks added for adaptive predicate ordering in apache#11262) was missing a few shapes that distinguish evaluation strategies for AND chains, and it had no result checking at all: 1. **An expensive-selective conjunct written *before* a cheap-unselective one.** The suite already had the reverse (`costsel` q03: cheap `c0 < 90` at ~90% first, expensive `regexp_like(s, 'rare')` at ~0.1% second). Without the mirror shape there is no case in the suite where the as-written order is already the optimal one, so the suite cannot tell "reordered well" apart from "reordered at all". It also exercises a distinct code path: measuring a *cheap* conjunct on the small survivor batch left after pre-selection by an expensive one, rather than measuring an expensive conjunct on a full batch. 2. **A many-column table.** Every existing `cardinality` query runs over the 16-column `ints` dataset, so the cost of materializing a filtered batch per level is small and roughly constant across the suite. Re-running the k = 8 predicate over a 64-column table separates the cost of *evaluating* the conjuncts from the cost of *filtering the batch* at each level, which is what per-level batch filtering actually pays for on wide inputs. 3. **No result validation.** The suite's template had no `result` directive, so `--result-mode validate` verified nothing on it. A reordering under test must not change *which* rows survive the filter, and every query here is a `count(*)` whose value is fixed by the generated data — a checked-in expected count is therefore a free correctness check on top of the timing. 4. **Nullable conjuncts.** Every predicate in the suite was non-nullable, so the suite never exercised the one case where writing the selective conjunct first buys nothing: `BinaryExpr`'s AND pre-selection bails out when the left-hand boolean array contains any NULL. 5. **Drift that a warm-up cannot absorb.** The existing `drift` dataset flipped which conjunct is selective late enough that a one-shot warm-up already lands on the post-flip order and is right for most of the scan. There was no shape where a decide-once strategy is wrong for a large part of the scan, and none at all where the *right* order differs between concurrently running partitions rather than over time. ## What changes are included in this PR? Everything is under `benchmarks/` (plus the suite's own docs); there are no Rust changes, and the suite discovers new `.benchmark` files automatically. - **Result validation.** The shared template gains `result sql_benchmarks/predicate_eval/results/${NAME}.csv`, and every benchmark's expected `count(*)` is checked in under `predicate_eval/results/`, so `--result-mode validate` now checks that a reordering under test still returns the same rows. The counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those; the `scale` and `width` subgroups pin their own values as template parameters (which win over the environment) and validate at any setting. The template's old `count(*) > 0` assert is dropped as redundant with this. - **`costsel` q04** — the mirror of q03: expensive-selective `regexp_like(s, 'rare')` written before cheap-unselective `c0 < 90`, so the as-written order is already the best one. q03/q04 bracket a reorderer when cost and selectivity point the same way. - **`cardinality` q34 and `load/ints_wide.sql`** — q32's k = 8 predicate over a 64-column table. `c0..c15` reuse `ints.sql`'s multipliers (all coprime to 100), so the hidden selectivities are identical to q32's and only the batch width changes. - **`nulls` subgroup (q90, q91).** `ints.sql` gains `c_sel`, NULL on 10% of rows and uniform on `[0,100)` elsewhere, so `c_sel < 5` is true on exactly 4% of rows. `check_short_circuit` returns `ShortCircuitStrategy::None` as soon as the left-hand boolean array has `null_count() > 0`, so the nullable selective conjunct gates nothing wherever it is written — q90 writes it first, q91 last. No `ints` query references `c_sel`, so their counts are unchanged. - **`drift` down to three cases.** q80 keeps the in-memory dataset, with the flip moved to 2% of the rows so it lands inside the pooled 8-batch warm-up (~65k rows at the default size). The mirrored-order twin was dropped as the same measurement. q82 and q83 share `load/drift_split.sql`: 16 Parquet files of `PRED_ROWS / 16` rows in a gitignored `predicate_eval/scratch/`, `f00..f07` with `a_sel = 0` selective (~0.1%) against `b_sel = 0` at ~50% and `f08..f15` the mirror. `FileGroup::split_files` sorts a group's files by path, so `target_partitions = 1` (q82) reads them in order and meets the flip halfway through one stream, and `= 16` (q83) gives every stream one whole file and one fixed profile — the case a decide-once or shared-decision strategy cannot get right. `repartition_file_scans = false` lives in the shared load file (it stops the groups being re-derived as byte ranges); `target_partitions` differs per query, so each `.benchmark` sets its own in an `init` step. The file names are fixed rather than written by one `COPY ... PARTITIONED BY`, because a directory-target `COPY` names its output with a random write id and Parquet has no overwrite, so the second load of the shared dataset would add files instead of replacing them. - **Shared queries for the knob sweeps.** The template takes a `QUERY` parameter (the path stem under `queries/`), so the `scale` and `width` stubs point at `costsel/q03` and `costsel/q01` rather than keeping seven verbatim copies of two WHERE clauses. - **Docs.** The `predicate_eval` row in `benchmarks/sql_benchmarks/README.md` carries the subgroup list, the result-validation behaviour with its defaults caveat, and the note that the suite sets no engine config of its own; `bench.sh` and the `.suite` description match. ## What is the testing strategy for this PR? No unit tests: these are benchmark definitions, not library code. Verification is that the suite parses, every query runs, and every query's result matches its checked-in expected count. A full run passes on a release build: `benchmark_runner predicate_eval -i 1 --result-mode validate` runs all 31 benchmarks across the ten subgroups and exits 0, and repeats cleanly (the drift `COPY`s overwrite their files rather than accumulating). Validation was checked to actually be checking: overwriting one expected count (`nulls_q90` 20000 → 999999) fails the run with ``` Error: Execution error: Error in result on row 1, column 1 running query "": expected value "999999" but got value "20000" in row: ["20000"] ``` and restoring the file makes it pass again; the same is true of `drift_q83_per_partition_skew` at 991 instead of 992. The drift plans are as intended on both sides (`EXPLAIN`, release build, file list capped at five entries): ``` -- q82, target_partitions = 1 FilterExec: a_sel@0 = 0 AND b_sel@1 = 0, projection=[] DataSourceExec: file_groups={1 group: [[.../drift_split/f00.parquet, .../f01.parquet, .../f02.parquet, .../f03.parquet, .../f04.parquet, ...]]}, projection=[a_sel, b_sel], file_type=parquet, predicate=a_sel@1 = 0 AND b_sel@2 = 0, ... -- q83, target_partitions = 16 FilterExec: a_sel@0 = 0 AND b_sel@1 = 0, projection=[] DataSourceExec: file_groups={16 groups: [[.../drift_split/f00.parquet], [.../f01.parquet], [.../f02.parquet], [.../f03.parquet], [.../f04.parquet], ...]}, projection=[a_sel, b_sel], file_type=parquet, predicate=a_sel@1 = 0 AND b_sel@2 = 0, ... ``` Both settings were checked to be load-bearing by dropping them: without q82's `init` the plan no longer contains `file_groups={1 group`, and with `repartition_file_scans` left on, q83 plans as `file_groups={12 groups: [[.../f00.parquet:0..224901, .../f01.parquet:0..74967], ...]}` — both profiles in one stream. That the read order is deterministic is what q82 relies on, and it is: `FileGroup::split_files` sorts a group's files by path before chunking them ([`file_groups.rs`](https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/file_groups.rs#L459-L462), whose comment says why: `ObjectStore::list` gives no order), and `FileStream` scans a group's files in order. The datasets were verified directly too: `c_sel` is NULL on exactly 100000 rows and `< 5` on 40000 at `PRED_ROWS=1000000`; the split dataset has 62500 rows per file, `a_sel = 0` on 62 of them and `b_sel = 0` on 31250 in `f00..f07`, mirrored in `f08..f15`, and 992 rows matching both over the whole table. q34 and q32 have identical selectivities (both `count(*) = 5000` at `PRED_ROWS=100000`). ## Are there any user-facing changes? No. Benchmark definitions only; nothing in this PR is part of any public API or affects query execution. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 14d18c4 commit 2a42140

108 files changed

Lines changed: 415 additions & 260 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎benchmarks/bench.sh‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ wide_schema: Small-projection queries on a wide synthetic dataset (10
106106
(runs both 'wide' and 'narrow' subgroups: narrow is an internal baseline; the wide-vs-narrow ratio is the signal)
107107
predicate_eval: Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an
108108
adaptive predicate-ordering system behaves across them (see https://github.com/apache/datafusion/issues/11262)
109-
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift)
109+
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift, nulls)
110110
(toggle a system under test with its native DATAFUSION_* env var; size data with PRED_ROWS, string width with PRED_FILL)
111111
parquet_row_filter_skip: Per-RG fully-matched RowFilter skip on Parquet (apache/datafusion#23696); clustered string key + low-selectivity
112112
range filter + pushdown, so most row groups are fully matched and the per-row RowFilter is skipped on them
@@ -876,7 +876,11 @@ run_push_down_topk() {
876876
# micro-benchmarks where each subgroup is a different predicate pattern, used to
877877
# test how an adaptive predicate-ordering system behaves across them (see
878878
# https://github.com/apache/datafusion/issues/11262). Data is generated inline
879-
# by the suite's load SQL, so there is no data step.
879+
# by the suite's load SQL, so there is no data step (drift q82 and q83 share 16
880+
# small Parquet files written into sql_benchmarks/predicate_eval/scratch/, which
881+
# is gitignored; q82 reads them with target_partitions=1, so the selectivity flip
882+
# lands halfway through one stream, and q83 with 16, so each stream gets one whole
883+
# file and a fixed profile).
880884
#
881885
# By default the suite measures DataFusion's built-in left-deep AND short-circuit
882886
# and sets no engine config of its own. To evaluate a system under test, export
@@ -885,7 +889,7 @@ run_push_down_topk() {
885889
# DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true ./bench.sh run predicate_eval
886890
# Suite-specific knobs (string-substituted into the load SQL, not engine config):
887891
# BENCH_SUBGROUP run one subgroup (costsel, cost, selectivity, cardinality,
888-
# width, scale, neutral, correlation, drift)
892+
# width, scale, neutral, correlation, drift, nulls)
889893
# PRED_ROWS synthetic row count (default 1_000_000; the scale subgroup
890894
# overrides this per query)
891895
# PRED_FILL filler chars per marker = string-column width knob

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ in the community:
4343
| `tpcds` | TPC‑DS queries |
4444
| `tpch` | TPC‑H queries |
4545
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46-
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Configure the system under test through its DataFusion settings. |
46+
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). |
4747
| `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. |
4848

4949
# Running Benchmarks
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup cardinality
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=cardinality
5-
QPAD=30
4+
QUERY=cardinality/q30
65
DATASET=ints
76
NAME=cardinality_q30_k2
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup cardinality
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=cardinality
5-
QPAD=31
4+
QUERY=cardinality/q31
65
DATASET=ints
76
NAME=cardinality_q31_k4
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup cardinality
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=cardinality
5-
QPAD=32
4+
QUERY=cardinality/q32
65
DATASET=ints
76
NAME=cardinality_q32_k8
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup cardinality
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=cardinality
5-
QPAD=33
4+
QUERY=cardinality/q33
65
DATASET=ints
76
NAME=cardinality_q33_k16
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
subgroup cardinality
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
QUERY=cardinality/q34
5+
DATASET=ints_wide
6+
NAME=cardinality_q34_k8_wide64
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup correlation
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=correlation
5-
QPAD=70
4+
QUERY=correlation/q70
65
DATASET=corr
76
NAME=correlation_q70_independent
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup correlation
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=correlation
5-
QPAD=71
4+
QUERY=correlation/q71
65
DATASET=corr
76
NAME=correlation_q71_positive
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
subgroup correlation
22

33
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4-
SUBGROUP=correlation
5-
QPAD=72
4+
QUERY=correlation/q72
65
DATASET=corr
76
NAME=correlation_q72_anti

0 commit comments

Comments
 (0)