Skip to content

Commit 903d349

Browse files
adriangbclaude
andcommitted
bench: simplify predicate_eval descriptions, link to apache#11262
Address review feedback: the suite/bench.sh/README descriptions were hard to parse. Lead with the point -- each subgroup is a different predicate pattern, used to test how an adaptive predicate-ordering system behaves across them -- and link to apache#11262 as an example of the change these benchmarks are meant to help drive, instead of inlining the full cost-axis breakdown. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent e478948 commit 903d349

8 files changed

Lines changed: 14 additions & 51 deletions

File tree

‎benchmarks/bench.sh‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,9 @@ push_down_topk: Benchmark of ORDER BY ... LIMIT over outer joins on TPC-
103103
external_aggr: External aggregation benchmark on TPC-H dataset (SF=1)
104104
wide_schema: Small-projection queries on a wide synthetic dataset (1024 cols × 256 files) — measures per-file metadata overhead
105105
(runs both 'wide' and 'narrow' subgroups: narrow is an internal baseline; the wide-vs-narrow ratio is the signal)
106-
predicate_eval: Implementation-agnostic conjunctive (AND) filter-evaluation micro-benchmarks (synthetic data, generated inline)
107-
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift, nulls)
106+
predicate_eval: Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an
107+
adaptive predicate-ordering system behaves across them (see https://github.com/apache/datafusion/issues/11262)
108+
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift)
108109
(toggle a system under test with its native DATAFUSION_* env var; size data with PRED_ROWS, string width with PRED_FILL)
109110
110111
# ClickBench Benchmarks
@@ -810,12 +811,15 @@ run_push_down_topk() {
810811
bash -c "$SQL_CARGO_COMMAND"
811812
}
812813

813-
# Runs the predicate_eval benchmark suite. Data is generated inline by the
814-
# suite's load SQL, so there is no data step. The suite is implementation-
815-
# agnostic and sets no engine config of its own; by default it measures
816-
# DataFusion's built-in left-deep AND short-circuit. To evaluate a
817-
# predicate-ordering system under test, export its native config as a
818-
# DATAFUSION_* env var before invoking bench.sh -- the bench harness reads
814+
# Runs the predicate_eval benchmark suite: conjunctive (AND) filter-evaluation
815+
# micro-benchmarks where each subgroup is a different predicate pattern, used to
816+
# test how an adaptive predicate-ordering system behaves across them (see
817+
# https://github.com/apache/datafusion/issues/11262). Data is generated inline
818+
# by the suite's load SQL, so there is no data step.
819+
#
820+
# By default the suite measures DataFusion's built-in left-deep AND short-circuit
821+
# and sets no engine config of its own. To evaluate a system under test, export
822+
# its native DATAFUSION_* config before invoking bench.sh -- the harness reads
819823
# SessionConfig::from_env, and that environment is inherited here, e.g.
820824
# DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true ./bench.sh run predicate_eval
821825
# Suite-specific knobs (string-substituted into the load SQL, not engine config):

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ in the community:
4343
| `tpcds` | TPC‑DS queries |
4444
| `tpch` | TPC‑H queries |
4545
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46-
| `predicate_eval` | Implementation-agnostic conjunctive (AND) filter-evaluation micro-benchmarks; subgroups (`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. Toggle a system under test with its native `DATAFUSION_*` env var |
46+
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Toggle a system under test with its native `DATAFUSION_*` env var |
4747

4848
# Running Benchmarks
4949

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q90.benchmark‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q91.benchmark‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

‎benchmarks/sql_benchmarks/predicate_eval/load/nulls.sql‎

Lines changed: 0 additions & 17 deletions
This file was deleted.
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
name = "predicate_eval"
2-
description = "Implementation-agnostic micro-benchmark suite for conjunctive (AND) filter evaluation. Characterises FilterExec along the axes that drive any predicate-ordering / AQE system: per-predicate cost and selectivity, the cost-weighted (cost/(1-sel)) interaction, conjunct count, string-column width, row count, predicate correlation, selectivity drift, nulls, and a pure-overhead control. Subgroups (select with BENCH_SUBGROUP): costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift, nulls. The suite measures DataFusion's built-in left-deep AND short-circuit by default and sets no engine config of its own; toggle a reordering system under test with its native DATAFUSION_* env var (the harness reads SessionConfig::from_env), e.g. DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true. Size synthetic data with PRED_ROWS and string-column width with PRED_FILL."
2+
description = "Micro-benchmarks for conjunctive (AND) filter evaluation. Each subgroup exercises a different predicate pattern (per-predicate cost, selectivity, conjunct count, string-column width, row count, correlation, selectivity drift, plus an order-neutral control) so the suite can show how an adaptive predicate-ordering system behaves across them -- the kind of change these benchmarks are meant to help drive, e.g. https://github.com/apache/datafusion/issues/11262. By default it measures DataFusion's built-in left-deep AND short-circuit and sets no engine config of its own; toggle a system under test with its native DATAFUSION_* env var (the harness reads SessionConfig::from_env), e.g. DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true. Subgroups (BENCH_SUBGROUP): costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift. Size synthetic data with PRED_ROWS and string-column width with PRED_FILL."

‎benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q90.sql‎

Lines changed: 0 additions & 5 deletions
This file was deleted.

‎benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q91.sql‎

Lines changed: 0 additions & 5 deletions
This file was deleted.

0 commit comments

Comments
 (0)