Skip to content

Commit 9307536

Browse files
claudeclaude
committed
bench(predicate_eval): smoke-test the real suite against small fixtures
The runner's tests build synthetic suites in a temporary directory, so nothing executes the checked-in predicate_eval files: a load script, a query, and an expected-count file can stop agreeing with each other and CI stays green. Add `predicate_eval_smoke`, which runs the real benchmark template for one shape per `load/*.sql` and validates the counts. The suite's `results/` fixtures were persisted at `PRED_ROWS=1000000`, far too much data for CI, and lowering the row count invalidates them, so the template's result path gains a `PRED_RESULTS` knob and the test points it at `results_smoke/`: the same shapes persisted at `PRED_ROWS=32000`, the smallest size that keeps every count non-zero (`drift_split` spreads the rows over 16 files and matches one row in 1000). Defaults are unchanged -- `PRED_RESULTS` falls back to `results`. The test also asserts that every load script is covered, so a new dataset cannot ship without a case. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NfKY4ipY99oSvjuTcP6aPX
1 parent de4864f commit 9307536

11 files changed

Lines changed: 201 additions & 2 deletions

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ in the community:
4343
| `tpcds` | TPC‑DS queries |
4444
| `tpch` | TPC‑H queries |
4545
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46-
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). |
46+
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). `PRED_RESULTS` picks a different directory of counts: `predicate_eval/results_smoke/` holds one shape per load script at `PRED_ROWS=32000`, which the `predicate_eval_smoke` test validates on every CI run so a load script, a query, and an expected-count file cannot drift apart unnoticed. |
4747
| `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. |
4848

4949
# Running Benchmarks
@@ -171,6 +171,7 @@ Some benchmarks use custom environment variables as outlined below:
171171
| SORTED_ORDER | Used in the clickbench_sorted benchmark to indicate the sort order of the column. | `ASC` |
172172
| PRED_ROWS | Used in the predicate_eval benchmark to size the synthetic table (the `scale` subgroup overrides this per query), and in the parquet_row_filter_skip benchmark to size the generated Parquet datasets (default `10000000` there). | `1000000` |
173173
| PRED_FILL | Used in the predicate_eval benchmark as the string-column width knob (filler chars per marker). | `30` |
174+
| PRED_RESULTS | Used in the predicate_eval benchmark to select the directory of expected counts, so a run at a row count other than the default can validate against its own fixtures. | `results` |
174175
| RG_SIZE | Used in the parquet_row_filter_skip benchmark as the Parquet `max_row_group_size` for the generated datasets. | `1000000` |
175176

176177
## How it works

‎benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.template‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,14 @@
66
# Optional, read by the load scripts as ${...:-default}:
77
# PRED_ROWS synthetic row count (default 1000000)
88
# PRED_FILL filler chars per marker, the string-width knob (default 30)
9+
# Optional, read here:
10+
# PRED_RESULTS results directory holding the expected counts (default results)
911
# The knob sweeps reuse another subgroup's query file through QUERY, and the table
1012
# is always named `t`, so cleanup is uniform across datasets.
13+
#
14+
# `results/` holds the counts at the suite defaults. `PRED_RESULTS=results_smoke`
15+
# selects the much smaller `PRED_ROWS=32000` counts that the `predicate_eval_smoke`
16+
# test validates, so a CI-sized run does not need the 1M-row fixtures.
1117

1218
load sql_benchmarks/predicate_eval/load/${DATASET}.sql
1319

@@ -16,6 +22,6 @@ group predicate_eval
1622

1723
run sql_benchmarks/predicate_eval/queries/${QUERY}.sql
1824

19-
result sql_benchmarks/predicate_eval/results/${NAME}.csv
25+
result sql_benchmarks/predicate_eval/${PRED_RESULTS:-results}/${NAME}.csv
2026

2127
cleanup sql_benchmarks/predicate_eval/init/cleanup.sql

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/cardinality_q34_k8_wide64.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
1600

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/correlation_q70_independent.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
1600

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/correlation_q73_redundant_proxy.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
2880

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/cost_q10_expensive_first.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
320

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/costsel_q01_regexp_selective_last.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
15

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/drift_q80_a_then_b.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
32

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/drift_q82_late_flip.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
32

‎benchmarks/sql_benchmarks/predicate_eval/results_smoke/nulls_q90_nullable_selective_first.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
count(*)
2+
640

0 commit comments

Comments
 (0)