Repository navigation
Use tpchgen-cli to generate tpch data in bench.sh
#19035
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
e0a9157
907bce3
21a0237
1790863
9a5a3b0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -246,25 +246,8 @@ You can enable `mimalloc` or `snmalloc` (to use either the mimalloc or snmalloc | |
| cargo run --release --features "mimalloc" --bin tpch -- benchmark datafusion --iterations 3 --path ./data --format tbl --query 1 --batch-size 4096 | ||
| ``` | ||
|
|
||
| The benchmark program also supports CSV and Parquet input file formats and a utility is provided to convert from `tbl` | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the data is generated directly using tpchgen-cli now, no need to convert data with another pass |
||
| (generated by the `dbgen` utility) to CSV and Parquet. | ||
|
|
||
| ```bash | ||
| cargo run --release --bin tpch -- convert --input ./data --output /mnt/tpch-parquet --format parquet | ||
| ``` | ||
|
|
||
| Or if you want to verify and run all the queries in the benchmark, you can just run `cargo test`. | ||
|
|
||
| #### Sorted Conversion | ||
|
|
||
| The TPCH tables generated by the dbgen utility are sorted by their first column (their primary key for most tables, the `l_orderkey` column for the `lineitem` table.) | ||
|
|
||
| To preserve this sorted order information during conversion (useful for benchmarking execution on pre-sorted data) include the `--sort` flag: | ||
|
|
||
| ```bash | ||
| cargo run --release --bin tpch -- convert --input ./data --output /mnt/tpch-sorted-parquet --format parquet --sort | ||
| ``` | ||
|
|
||
| ### Comparing results between runs | ||
|
|
||
| Any `dfbench` execution with `-o <dir>` argument will produce a | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -203,26 +203,22 @@ main() { | |||||
| # nlj uses range() function, no data generation needed | ||||||
| ;; | ||||||
| tpch) | ||||||
| data_tpch "1" | ||||||
| data_tpch "1" "parquet" | ||||||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. per @comphead 's suggestion, now only the format required is made, rather than always creating all three formats |
||||||
| ;; | ||||||
| tpch_mem) | ||||||
| # same data as for tpch | ||||||
| data_tpch "1" | ||||||
| data_tpch "1" "parquet" | ||||||
| ;; | ||||||
| tpch_csv) | ||||||
| # same data as for tpch | ||||||
| data_tpch "1" | ||||||
| data_tpch "1" "csv" | ||||||
| ;; | ||||||
| tpch10) | ||||||
| data_tpch "10" | ||||||
| data_tpch "10" "parquet" | ||||||
| ;; | ||||||
| tpch_mem10) | ||||||
| # same data as for tpch10 | ||||||
| data_tpch "10" | ||||||
| data_tpch "10" "parquet" | ||||||
| ;; | ||||||
| tpch_csv10) | ||||||
| # same data as for tpch10 | ||||||
| data_tpch "10" | ||||||
| data_tpch "10" "csv" | ||||||
| ;; | ||||||
| clickbench_1) | ||||||
| data_clickbench_1 | ||||||
|
|
@@ -537,7 +533,7 @@ main() { | |||||
| # Creates TPCH data at a certain scale factor, if it doesn't already | ||||||
| # exist | ||||||
| # | ||||||
| # call like: data_tpch($scale_factor) | ||||||
| # call like: data_tpch($scale_factor, format) | ||||||
| # | ||||||
| # Creates data in $DATA_DIR/tpch_sf1 for scale factor 1 | ||||||
| # Creates data in $DATA_DIR/tpch_sf10 for scale factor 10 | ||||||
|
|
@@ -548,20 +544,19 @@ data_tpch() { | |||||
| echo "Internal error: Scale factor not specified" | ||||||
| exit 1 | ||||||
| fi | ||||||
| FORMAT=$2 | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As I understand, that would default the argument to
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See #19035 (comment) https://github.com/apache/datafusion/pull/19035/files/907bce3e16352148eade3b7cf512091a9aab4232#diff-1769f5787dc11c8b1f1b48288cdf3c89d25a5b5cbc6be4740bfcc70a6313ba99R550 will print And the third reason why I proposed This sounds like parquet is the needed format most of the time. But data_h2o() uses CSV as a default format:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, I see what you are saying now - there are several other calls to data_tpch that don't pass the format argument here eg sort_tpch)
# same data as for tpch
data_tpch "1"
;;
sort_tpch10)
# same data as for tpch10
data_tpch "10"
;;
topk_tpch)
# same data as for tpch
data_tpch "1"
;;
nlj)I will update those calls to explicitly pass in a format as I think that will make it clear what is going on
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. in 9a5a3b0
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍🏻 |
||||||
|
|
||||||
| TPCH_DIR="${DATA_DIR}/tpch_sf${SCALE_FACTOR}" | ||||||
| echo "Creating tpch dataset at Scale Factor ${SCALE_FACTOR} in ${TPCH_DIR}..." | ||||||
| echo "Creating tpch $FORMAT dataset at Scale Factor ${SCALE_FACTOR} in ${TPCH_DIR}..." | ||||||
|
|
||||||
| # Ensure the target data directory exists | ||||||
| mkdir -p "${TPCH_DIR}" | ||||||
|
|
||||||
| # Create 'tbl' (CSV format) data into $DATA_DIR if it does not already exist | ||||||
| FILE="${TPCH_DIR}/supplier.tbl" | ||||||
| if test -f "${FILE}"; then | ||||||
| echo " tbl files exist ($FILE exists)." | ||||||
| else | ||||||
| echo " creating tbl files with tpch_dbgen..." | ||||||
| docker run -v "${TPCH_DIR}":/data -it --rm ghcr.io/scalytics/tpch-docker:main -vf -s "${SCALE_FACTOR}" | ||||||
| # check if tpchgen_cli is installed | ||||||
|
alamb marked this conversation as resolved.
Resolved
|
||||||
| if ! command -v tpchgen-cli &> /dev/null | ||||||
| then | ||||||
| echo "tpchgen_cli could not be found, please install it via 'cargo install tpchgen-cli'" | ||||||
|
alamb marked this conversation as resolved.
Resolved
|
||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # Copy expected answers into the ./data/answers directory if it does not already exist | ||||||
|
|
@@ -574,27 +569,32 @@ data_tpch() { | |||||
| docker run -v "${TPCH_DIR}":/data -it --entrypoint /bin/bash --rm ghcr.io/scalytics/tpch-docker:main -c "cp -f /opt/tpch/2.18.0_rc2/dbgen/answers/* /data/answers/" | ||||||
| fi | ||||||
|
|
||||||
| # Create 'parquet' files from tbl | ||||||
| FILE="${TPCH_DIR}/supplier" | ||||||
| if test -d "${FILE}"; then | ||||||
| echo " parquet files exist ($FILE exists)." | ||||||
| else | ||||||
| echo " creating parquet files using benchmark binary ..." | ||||||
| pushd "${SCRIPT_DIR}" > /dev/null | ||||||
| $CARGO_COMMAND --bin tpch -- convert --input "${TPCH_DIR}" --output "${TPCH_DIR}" --format parquet | ||||||
| popd > /dev/null | ||||||
| if [ "$FORMAT" = "parquet" ]; then | ||||||
| # Create 'parquet' files, one directory per file | ||||||
| FILE="${TPCH_DIR}/supplier" | ||||||
| if test -d "${FILE}"; then | ||||||
| echo " parquet files exist ($FILE exists)." | ||||||
| else | ||||||
| echo " creating parquet files using tpchgen-cli ..." | ||||||
| tpchgen-cli --scale-factor "${SCALE_FACTOR}" --format parquet --parquet-compression='ZSTD(1)' --parts=1 --output-dir "${TPCH_DIR}" | ||||||
|
alamb marked this conversation as resolved.
Resolved
|
||||||
| fi | ||||||
| return | ||||||
| fi | ||||||
|
|
||||||
| # Create 'csv' files from tbl | ||||||
| FILE="${TPCH_DIR}/csv/supplier" | ||||||
| if test -d "${FILE}"; then | ||||||
| echo " csv files exist ($FILE exists)." | ||||||
| else | ||||||
| echo " creating csv files using benchmark binary ..." | ||||||
| pushd "${SCRIPT_DIR}" > /dev/null | ||||||
| $CARGO_COMMAND --bin tpch -- convert --input "${TPCH_DIR}" --output "${TPCH_DIR}/csv" --format csv | ||||||
| popd > /dev/null | ||||||
| # Create 'csv' files, one directory per file | ||||||
| if [ "$FORMAT" = "csv" ]; then | ||||||
| FILE="${TPCH_DIR}/csv/supplier" | ||||||
| if test -d "${FILE}"; then | ||||||
| echo " csv files exist ($FILE exists)." | ||||||
| else | ||||||
| echo " creating csv files using tpchgen-cli binary ..." | ||||||
| tpchgen-cli --scale-factor "${SCALE_FACTOR}" --format csv --parts=1 --output-dir "${TPCH_DIR}/csv" | ||||||
| fi | ||||||
| return | ||||||
| fi | ||||||
|
|
||||||
| echo "Error: unknown format '$FORMAT' for tpch data generation, expected 'parquet' or 'csv'" | ||||||
| exit 1 | ||||||
| } | ||||||
|
|
||||||
| # Runs the tpch benchmark | ||||||
|
|
@@ -611,10 +611,10 @@ run_tpch() { | |||||
| echo "Running tpch benchmark..." | ||||||
|
|
||||||
| FORMAT=$2 | ||||||
|
alamb marked this conversation as resolved.
Resolved
|
||||||
| debug_run $CARGO_COMMAND --bin tpch -- benchmark datafusion --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" --format ${FORMAT} -o "${RESULTS_FILE}" ${QUERY_ARG} | ||||||
| debug_run $CARGO_COMMAND --bin dfbench -- tpch --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" --format ${FORMAT} -o "${RESULTS_FILE}" ${QUERY_ARG} | ||||||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this updates bench.sh to use dfbench directly rather than the alternate |
||||||
| } | ||||||
|
|
||||||
| # Runs the tpch in memory | ||||||
| # Runs the tpch in memory (needs tpch parquet data) | ||||||
| run_tpch_mem() { | ||||||
| SCALE_FACTOR=$1 | ||||||
| if [ -z "$SCALE_FACTOR" ] ; then | ||||||
|
|
@@ -627,7 +627,7 @@ run_tpch_mem() { | |||||
| echo "RESULTS_FILE: ${RESULTS_FILE}" | ||||||
| echo "Running tpch_mem benchmark..." | ||||||
| # -m means in memory | ||||||
| debug_run $CARGO_COMMAND --bin tpch -- benchmark datafusion --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" -m --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG} | ||||||
| debug_run $CARGO_COMMAND --bin dfbench -- tpch --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" -m --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG} | ||||||
| } | ||||||
|
|
||||||
| # Runs the compile profile benchmark helper | ||||||
|
|
||||||
This file was deleted.
Uh oh!
There was an error while loading. Please reload this page.