Repository navigation
Commit 2a42140
bench(predicate_eval): add expensive-first and wide-column shapes (apache#25032)
## Which issue does this PR close?
- Part of apache#22883
- Related to apache#11262
- Prequel split out of apache#22698: these are the benchmark-suite additions
from that PR, carried on their own so they can land (and be run)
independently of the adaptive-filter work.
## Rationale for this change
The `predicate_eval` suite (the conjunctive-filter microbenchmarks added
for adaptive predicate ordering in apache#11262) was missing a few shapes that
distinguish evaluation strategies for AND chains, and it had no result
checking at all:
1. **An expensive-selective conjunct written *before* a
cheap-unselective one.** The suite already had the reverse (`costsel`
q03: cheap `c0 < 90` at ~90% first, expensive `regexp_like(s, 'rare')`
at ~0.1% second). Without the mirror shape there is no case in the suite
where the as-written order is already the optimal one, so the suite
cannot tell "reordered well" apart from "reordered at all". It also
exercises a distinct code path: measuring a *cheap* conjunct on the
small survivor batch left after pre-selection by an expensive one,
rather than measuring an expensive conjunct on a full batch.
2. **A many-column table.** Every existing `cardinality` query runs over
the 16-column `ints` dataset, so the cost of materializing a filtered
batch per level is small and roughly constant across the suite.
Re-running the k = 8 predicate over a 64-column table separates the cost
of *evaluating* the conjuncts from the cost of *filtering the batch* at
each level, which is what per-level batch filtering actually pays for on
wide inputs.
3. **No result validation.** The suite's template had no `result`
directive, so `--result-mode validate` verified nothing on it. A
reordering under test must not change *which* rows survive the filter,
and every query here is a `count(*)` whose value is fixed by the
generated data — a checked-in expected count is therefore a free
correctness check on top of the timing.
4. **Nullable conjuncts.** Every predicate in the suite was
non-nullable, so the suite never exercised the one case where writing
the selective conjunct first buys nothing: `BinaryExpr`'s AND
pre-selection bails out when the left-hand boolean array contains any
NULL.
5. **Drift that a warm-up cannot absorb.** The existing `drift` dataset
flipped which conjunct is selective late enough that a one-shot warm-up
already lands on the post-flip order and is right for most of the scan.
There was no shape where a decide-once strategy is wrong for a large
part of the scan, and none at all where the *right* order differs
between concurrently running partitions rather than over time.
## What changes are included in this PR?
Everything is under `benchmarks/` (plus the suite's own docs); there are
no Rust changes, and the suite discovers new `.benchmark` files
automatically.
- **Result validation.** The shared template gains `result
sql_benchmarks/predicate_eval/results/${NAME}.csv`, and every
benchmark's expected `count(*)` is checked in under
`predicate_eval/results/`, so `--result-mode validate` now checks that a
reordering under test still returns the same rows. The counts were
persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`),
so validation assumes those; the `scale` and `width` subgroups pin their
own values as template parameters (which win over the environment) and
validate at any setting. The template's old `count(*) > 0` assert is
dropped as redundant with this.
- **`costsel` q04** — the mirror of q03: expensive-selective
`regexp_like(s, 'rare')` written before cheap-unselective `c0 < 90`, so
the as-written order is already the best one. q03/q04 bracket a
reorderer when cost and selectivity point the same way.
- **`cardinality` q34 and `load/ints_wide.sql`** — q32's k = 8 predicate
over a 64-column table. `c0..c15` reuse `ints.sql`'s multipliers (all
coprime to 100), so the hidden selectivities are identical to q32's and
only the batch width changes.
- **`nulls` subgroup (q90, q91).** `ints.sql` gains `c_sel`, NULL on 10%
of rows and uniform on `[0,100)` elsewhere, so `c_sel < 5` is true on
exactly 4% of rows. `check_short_circuit` returns
`ShortCircuitStrategy::None` as soon as the left-hand boolean array has
`null_count() > 0`, so the nullable selective conjunct gates nothing
wherever it is written — q90 writes it first, q91 last. No `ints` query
references `c_sel`, so their counts are unchanged.
- **`drift` down to three cases.** q80 keeps the in-memory dataset, with
the flip moved to 2% of the rows so it lands inside the pooled 8-batch
warm-up (~65k rows at the default size). The mirrored-order twin was
dropped as the same measurement. q82 and q83 share
`load/drift_split.sql`: 16 Parquet files of `PRED_ROWS / 16` rows in a
gitignored `predicate_eval/scratch/`, `f00..f07` with `a_sel = 0`
selective (~0.1%) against `b_sel = 0` at ~50% and `f08..f15` the mirror.
`FileGroup::split_files` sorts a group's files by path, so
`target_partitions = 1` (q82) reads them in order and meets the flip
halfway through one stream, and `= 16` (q83) gives every stream one
whole file and one fixed profile — the case a decide-once or
shared-decision strategy cannot get right. `repartition_file_scans =
false` lives in the shared load file (it stops the groups being
re-derived as byte ranges); `target_partitions` differs per query, so
each `.benchmark` sets its own in an `init` step. The file names are
fixed rather than written by one `COPY ... PARTITIONED BY`, because a
directory-target `COPY` names its output with a random write id and
Parquet has no overwrite, so the second load of the shared dataset would
add files instead of replacing them.
- **Shared queries for the knob sweeps.** The template takes a `QUERY`
parameter (the path stem under `queries/`), so the `scale` and `width`
stubs point at `costsel/q03` and `costsel/q01` rather than keeping seven
verbatim copies of two WHERE clauses.
- **Docs.** The `predicate_eval` row in
`benchmarks/sql_benchmarks/README.md` carries the subgroup list, the
result-validation behaviour with its defaults caveat, and the note that
the suite sets no engine config of its own; `bench.sh` and the `.suite`
description match.
## What is the testing strategy for this PR?
No unit tests: these are benchmark definitions, not library code.
Verification is that the suite parses, every query runs, and every
query's result matches its checked-in expected count.
A full run passes on a release build: `benchmark_runner predicate_eval
-i 1 --result-mode validate` runs all 31 benchmarks across the ten
subgroups and exits 0, and repeats cleanly (the drift `COPY`s overwrite
their files rather than accumulating).
Validation was checked to actually be checking: overwriting one expected
count (`nulls_q90` 20000 → 999999) fails the run with
```
Error: Execution error: Error in result on row 1, column 1 running query "": expected value "999999" but got value "20000" in row: ["20000"]
```
and restoring the file makes it pass again; the same is true of
`drift_q83_per_partition_skew` at 991 instead of 992.
The drift plans are as intended on both sides (`EXPLAIN`, release build,
file list capped at five entries):
```
-- q82, target_partitions = 1
FilterExec: a_sel@0 = 0 AND b_sel@1 = 0, projection=[]
DataSourceExec: file_groups={1 group: [[.../drift_split/f00.parquet, .../f01.parquet, .../f02.parquet, .../f03.parquet, .../f04.parquet, ...]]}, projection=[a_sel, b_sel], file_type=parquet, predicate=a_sel@1 = 0 AND b_sel@2 = 0, ...
-- q83, target_partitions = 16
FilterExec: a_sel@0 = 0 AND b_sel@1 = 0, projection=[]
DataSourceExec: file_groups={16 groups: [[.../drift_split/f00.parquet], [.../f01.parquet], [.../f02.parquet], [.../f03.parquet], [.../f04.parquet], ...]}, projection=[a_sel, b_sel], file_type=parquet, predicate=a_sel@1 = 0 AND b_sel@2 = 0, ...
```
Both settings were checked to be load-bearing by dropping them: without
q82's `init` the plan no longer contains `file_groups={1 group`, and
with `repartition_file_scans` left on, q83 plans as `file_groups={12
groups: [[.../f00.parquet:0..224901, .../f01.parquet:0..74967], ...]}` —
both profiles in one stream. That the read order is deterministic is
what q82 relies on, and it is: `FileGroup::split_files` sorts a group's
files by path before chunking them
([`file_groups.rs`](https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/file_groups.rs#L459-L462),
whose comment says why: `ObjectStore::list` gives no order), and
`FileStream` scans a group's files in order.
The datasets were verified directly too: `c_sel` is NULL on exactly
100000 rows and `< 5` on 40000 at `PRED_ROWS=1000000`; the split dataset
has 62500 rows per file, `a_sel = 0` on 62 of them and `b_sel = 0` on
31250 in `f00..f07`, mirrored in `f08..f15`, and 992 rows matching both
over the whole table. q34 and q32 have identical selectivities (both
`count(*) = 5000` at `PRED_ROWS=100000`).
## Are there any user-facing changes?
No. Benchmark definitions only; nothing in this PR is part of any public
API or affects query execution.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.1 <[email protected]>1 parent 14d18c4 commit 2a42140
108 files changed
Lines changed: 415 additions & 260 deletions
File tree
- benchmarks
- sql_benchmarks
- predicate_eval
- benchmarks
- cardinality
- correlation
- costsel
- cost
- drift
- neutral
- nulls
- scale
- selectivity
- width
- load
- queries
- cardinality
- correlation
- costsel
- cost
- drift
- neutral
- nulls
- scale
- selectivity
- width
- results
- scratch
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
109 | | - | |
| 109 | + | |
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
| |||
876 | 876 | | |
877 | 877 | | |
878 | 878 | | |
879 | | - | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
880 | 884 | | |
881 | 885 | | |
882 | 886 | | |
| |||
885 | 889 | | |
886 | 890 | | |
887 | 891 | | |
888 | | - | |
| 892 | + | |
889 | 893 | | |
890 | 894 | | |
891 | 895 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
6 | 5 | | |
7 | 6 | | |
0 commit comments