Test program that spawns n number of tokio task with given sql query on datafusion. For experimental use.
Clone apache-datafusion repo
Initalize project submodules
git submodule update --init --recursiveThis section is taken from arrow datafusion parquet benchmarks
Run the following test command to generate data. This data will usually at data/logs.parquet in the parquet-testing
cd parquet-testing
cargo run --release --bin parquet -- filter --path ./data --scale-factor 1.0As the benchmark is running you can copy over data/logs.parquet and close the running process.
Clone this repo and move over / configure test parquet file
then run the following
cargo r -- <count> "sql" | less -Schange count to set number of concurrent requests
| ❗ Important |
|---|
| Do not specify large count number for the concurrent queries. |
For the query itself add EXPLAIN ANALYZE to output metrics instead of result |
---------------
OS: Arch Linux x86_64
CPU: AMD Ryzen 5 1600 (12) @ 3.200GHz
Memory: 4778MiB / 7888MiB
cargo r -- 1 "explain analyze select * from test where request_bytes >= 15000"
Time taken: 7
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=4878770, elapsed_compute=3.645135266s] |
| | FilterExec: request_bytes@11 >= 15000, metrics=[output_rows=4878770, elapsed_compute=7.781150113s] |
| | ParquetExec: file_groups={12 groups: [[home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:0..101900044], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:101900044..203800088], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:203800088..305700132], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:305700132..407600176], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:407600176..509500220], ...]}, projection=[service, host, pod, container, image, time, client_addr, request_duration_ns, request_user_agent, request_method, request_host, request_bytes, response_bytes, response_status, decimal_price], predicate=request_bytes@11 >= 15000, pruning_predicate=request_bytes_max@0 >= 15000, metrics=[output_rows=10838832, elapsed_compute=12ns, file_open_errors=0, pushdown_rows_filtered=0, bytes_scanned=1223934655, row_groups_pruned=0, num_predicate_creation_errors=0, page_index_rows_filtered=0, file_scan_errors=0, predicate_evaluation_errors=0, page_index_eval_time=1.169156ms, time_elapsed_scanning_until_data=6.400948734s, time_elapsed_opening=235.312585ms, time_elapsed_processing=53.645110109s, pushdown_eval_time=24ns, time_elapsed_scanning_total=70.193552261s] |
| | |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
cargo r -- 3 "explain analyze select * from test where request_bytes >= 15000"
Time taken: 15
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=4878770, elapsed_compute=4.247010423s] |
| | FilterExec: request_bytes@11 >= 15000, metrics=[output_rows=4878770, elapsed_compute=8.972745353s] |
| | ParquetExec: file_groups={12 groups: [[home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:0..101900044], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:101900044..203800088], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:203800088..305700132], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:305700132..407600176], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:407600176..509500220], ...]}, projection=[service, host, pod, container, image, time, client_addr, request_duration_ns, request_user_agent, request_method, request_host, request_bytes, response_bytes, response_status, decimal_price], predicate=request_bytes@11 >= 15000, pruning_predicate=request_bytes_max@0 >= 15000, metrics=[output_rows=10838832, elapsed_compute=12ns, file_scan_errors=0, row_groups_pruned=0, file_open_errors=0, predicate_evaluation_errors=0, bytes_scanned=1223934655, page_index_rows_filtered=0, num_predicate_creation_errors=0, pushdown_rows_filtered=0, time_elapsed_scanning_total=104.355523151s, time_elapsed_scanning_until_data=32.997534227s, pushdown_eval_time=24ns, time_elapsed_processing=60.289838687s, time_elapsed_opening=353.7616ms, page_index_eval_time=873.26µs] |
| | |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Time taken: 20
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=4878770, elapsed_compute=3.782139462s] |
| | FilterExec: request_bytes@11 >= 15000, metrics=[output_rows=4878770, elapsed_compute=8.040142398s] |
| | ParquetExec: file_groups={12 groups: [[home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:0..101900044], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:101900044..203800088], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:203800088..305700132], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:305700132..407600176], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:407600176..509500220], ...]}, projection=[service, host, pod, container, image, time, client_addr, request_duration_ns, request_user_agent, request_method, request_host, request_bytes, response_bytes, response_status, decimal_price], predicate=request_bytes@11 >= 15000, pruning_predicate=request_bytes_max@0 >= 15000, metrics=[output_rows=10838832, elapsed_compute=12ns, file_scan_errors=0, row_groups_pruned=0, file_open_errors=0, predicate_evaluation_errors=0, bytes_scanned=1223934655, page_index_rows_filtered=0, num_predicate_creation_errors=0, pushdown_rows_filtered=0, time_elapsed_scanning_total=158.165254331s, time_elapsed_scanning_until_data=92.896547706s, pushdown_eval_time=24ns, time_elapsed_processing=55.253859469s, time_elapsed_opening=803.854171ms, page_index_eval_time=1.060972ms] |
| | |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Time taken: 20
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=4878770, elapsed_compute=3.795169738s] |
| | FilterExec: request_bytes@11 >= 15000, metrics=[output_rows=4878770, elapsed_compute=8.101052806s] |
| | ParquetExec: file_groups={12 groups: [[home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:0..101900044], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:101900044..203800088], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:203800088..305700132], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:305700132..407600176], [home/trueleo/datafusion benchmark/concurrent_test_datafusion/logs.parquet:407600176..509500220], ...]}, projection=[service, host, pod, container, image, time, client_addr, request_duration_ns, request_user_agent, request_method, request_host, request_bytes, response_bytes, response_status, decimal_price], predicate=request_bytes@11 >= 15000, pruning_predicate=request_bytes_max@0 >= 15000, metrics=[output_rows=10838832, elapsed_compute=12ns, file_scan_errors=0, row_groups_pruned=0, file_open_errors=0, predicate_evaluation_errors=0, bytes_scanned=1223934655, page_index_rows_filtered=0, num_predicate_creation_errors=0, pushdown_rows_filtered=0, time_elapsed_scanning_total=161.998678133s, time_elapsed_scanning_until_data=96.881614644s, pushdown_eval_time=24ns, time_elapsed_processing=54.905152361s, time_elapsed_opening=783.138398ms, page_index_eval_time=964.673µs] |
| | |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+