Skip to content

Commit 9b81ff8

Browse files
adriangbclaude
adriangb
and
claude
authored
bench: make wide_schema honor DATA_DIR like the other sql_benchmarks (#22836)
## Which issue does this PR close? - Relates to #21968 (wide-schema benchmark work). ## Rationale for this change Every `sql_benchmarks` suite declares its external-table `LOCATION` as `${DATA_DIR:-data}/...`, and the matching `bench.sh run_*` forwards `DATA_DIR="${DATA_DIR}"` to the benchmark process — e.g. `clickbench`, `imdb`, `tpch`, `push_down_topk`, `h2o` all do this. `wide_schema` (added in #21970) diverged on **both** counts: - `wide_schema/init/load.sql` hardcoded `LOCATION 'data/wide_schema/...'` (no `${DATA_DIR}`) - `run_wide_schema` did not forward `DATA_DIR` Run from inside the repo this happens to work, because the process CWD is the repo's `benchmarks/` and the relative `data/` path resolves. But it breaks whenever the data directory is not under the current directory — for example the CI benchmark runner builds/runs the benchmark from a separate source checkout while staging the dataset elsewhere and pointing at it via `DATA_DIR`. There, wide_schema fails at load: ``` initialization failed: No files found at file:///.../benchmarks/data/wide_schema/wide/. Cannot infer schema from an empty location; either add data files or declare an explicit schema for the table. ``` (The `bench.sh` variable `DATA_DIR` is set but not exported, so it only reaches the benchmark when a `run_*` function passes it explicitly — which `run_tpch` etc. do and `run_wide_schema` did not.) ## What changes are included in this PR? Two one-line fixes bringing `wide_schema` in line with the convention used by every other suite: - `load.sql`: `data/wide_schema/...` → `${DATA_DIR:-data}/wide_schema/...` - `run_wide_schema`: forward `DATA_DIR="${DATA_DIR}"` for both the `wide` and `narrow` subgroups ## Are these changes tested? Benchmark tooling only. The substitution path is the same one every other suite already relies on: the SQL harness's `process_replacements` resolves `${DATA_DIR:-data}` from the env (falling back to `data`), and `run_wide_schema` now forwards `DATA_DIR` exactly as `run_tpch` does. Verified `bench.sh` still parses (`bash -n`). Default in-repo behavior is unchanged (CWD-relative `data/`). ## Are there any user-facing changes? No. Benchmark harness only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent bdfdd09 commit 9b81ff8

2 files changed

Lines changed: 3 additions & 1 deletion

File tree

‎benchmarks/bench.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -786,12 +786,14 @@ data_wide_schema() {
786786
run_wide_schema() {
787787
echo "Running wide_schema benchmark (wide subgroup)..."
788788
debug_run env BENCH_NAME=wide_schema BENCH_SUBGROUP=wide \
789+
DATA_DIR="${DATA_DIR}" \
789790
SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
790791
${QUERY:+BENCH_QUERY="${QUERY}"} \
791792
bash -c "$SQL_CARGO_COMMAND"
792793

793794
echo "Running wide_schema benchmark (narrow baseline subgroup)..."
794795
debug_run env BENCH_NAME=wide_schema BENCH_SUBGROUP=narrow \
796+
DATA_DIR="${DATA_DIR}" \
795797
SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
796798
${QUERY:+BENCH_QUERY="${QUERY}"} \
797799
bash -c "$SQL_CARGO_COMMAND"

‎benchmarks/sql_benchmarks/wide_schema/init/load.sql‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33
-- BENCH_SUBGROUP=wide → 1024-col synthetic dataset (the actual benchmark)
44
-- BENCH_SUBGROUP=narrow → 8-col baseline (companion only — meaningful
55
-- only when compared to the wide numbers)
6-
CREATE EXTERNAL TABLE events STORED AS PARQUET LOCATION 'data/wide_schema/${BENCH_SUBGROUP:-wide}/';
6+
CREATE EXTERNAL TABLE events STORED AS PARQUET LOCATION '${DATA_DIR:-data}/wide_schema/${BENCH_SUBGROUP:-wide}/';

0 commit comments

Comments
 (0)