Skip to content

Commit 83022b1

Browse files
authored
SQL benchmark configuration parity updates - apache#24778 2 (apache#24803)
## Which issue does this PR close? Part of apache#21706. This PR does not close the issue. ## Rationale for this change The SQL-based TPC-H, TPC-DS, and sort TPC-H benchmarks did not consistently expose the same schemas, constraints, and loading behavior as their native Rust equivalents. These differences could affect query planning and make benchmark results less directly comparable. ## What changes are included in this PR? - Align the TPC-H CSV, Parquet, and in-memory table definitions with the native benchmark's data types, nullability, and primary-key constraints. - Add explicit TPC-DS schemas and primary-key constraints while retaining the physical types used by the DataFusion benchmark Parquet dataset. - Update the sort TPC-H loader to: - Read the complete `lineitem` Parquet directory rather than only `lineitem.1.parquet`. - Keep the table Parquet-backed instead of materializing it. - Declare existing `l_orderkey ASC NULLS LAST` ordering as metadata instead of physically sorting the data. The sort TPC-H changes match the default behavior of the native Rust benchmark when it is run without `--mem-table`. ## Are these changes tested? Yes. The following checks were run: ```shell cargo fmt --all -- --check cargo clippy --all-targets --all-features -- -D warnings Sort TPC-H query 1 was also executed against the SF1 dataset in both modes: cargo run --profile release -p datafusion-benchmarks \ --bin benchmark_runner -- sort_tpch \ --scale-factor 1 --sorted false --query 1 --iterations 1 cargo run --profile release -p datafusion-benchmarks \ --bin benchmark_runner -- sort_tpch \ --scale-factor 1 --sorted true --query 1 --iterations 1 Both modes completed successfully and returned all 6,001,215 lineitem rows. ## Are there any user-facing changes? No public API behavior changes. The SQL benchmark definitions now more closely match the corresponding native benchmark configurations.
1 parent 07483c1 commit 83022b1

5 files changed

Lines changed: 775 additions & 105 deletions

File tree

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1-
CREATE EXTERNAL TABLE lineitem_raw STORED AS PARQUET LOCATION '${DATA_DIR:-data}/tpch_sf${BENCH_SIZE:-1}/lineitem/lineitem.1.parquet';
2-
3-
CREATE TABLE lineitem as (SELECT * FROM lineitem_raw${BENCH_SORTED:-false| order by l_orderkey asc| });
1+
CREATE EXTERNAL TABLE lineitem
2+
STORED AS PARQUET
3+
LOCATION '${DATA_DIR:-data}/tpch_sf${BENCH_SIZE:-1}/lineitem'
4+
${BENCH_SORTED:-false|WITH ORDER (l_orderkey ASC NULLS LAST)| };

0 commit comments

Comments
 (0)