You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d4c6299
Browse filesBrowse the repository at this point in the historyBrowse files
datafusion.execution.enable_nlj_coordinated_fallback true Enables the memory-limited fallback for `NestedLoopJoinExec` join types that emit unmatched left rows in the final output (LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK, FULL) when the right side has multiple partitions. This fallback coordinates per-chunk left state (visited bitmap and probe-thread counter) across all right-side partitions, which assumes every partition runs in the same process. Distributed engines that execute each output partition as an independent task (e.g. Ballista, datafusion-distributed) build a separate coordinator per task and poll only one partition, so the cross-partition counter never reaches zero and the fallback would stall. Such engines should set this to `false`: the coordinated fallback is then disabled for left-emitting multi-partition joins, which instead fail with a resource-exhaustion error under memory pressure rather than deadlocking. Single-partition and non-left-emitting joins are unaffected and always keep the fallback.
385
385
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
386
386
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower. Note: this option currently only applies to the symmetric hash join.
387
-
datafusion.execution.hash_aggregate_bucket_threshold 0 (experimental) Number of groups above which a hash aggregation stops growing a single hash table, so that its tables stay small enough to be cache friendly. A partial aggregation then emits the state of its table and starts over, as long as the emitted groups do not come back. A final aggregation splits the groups seen so far and all further input into hash buckets, which are aggregated one after another and can be spilled and released independently. Set to 0 to disable.
387
+
datafusion.execution.hash_aggregate_bucket_threshold 0 (experimental) Number of groups above which a hash aggregation stops growing a single hash table, so that its tables stay small enough to be cache friendly. A partial aggregation then emits the state of its table and starts over, as long as the emitted groups do not come back. A final aggregation splits the groups seen so far and all further input into hash buckets, which are aggregated one after another and can be spilled and released independently; it does so at a quarter of this number when its input holds about one row per group. Aggregations of millions of groups per partition run faster and with less memory. Moving rows into buckets has a cost of its own, so aggregations that end at a few times this number of groups, string keys in particular, can run a few percent slower, and input that repeats its groups can use more memory. Set to 0 to disable.
388
388
datafusion.execution.hash_join_buffering_capacity 0 How many bytes to buffer in the probe side of hash joins while the build side is concurrently being built. Without this, hash joins will wait until the full materialization of the build side before polling the probe side. This is useful in scenarios where the query is not completely CPU bounded, allowing to do some early work concurrently and reducing the latency of the query. Note that when hash join buffering is enabled, the probe side will start eagerly polling data, not giving time for the producer side of dynamic filters to produce any meaningful predicate. Queries with dynamic filters might see performance degradation. Disabled by default, set to a number greater than 0 for enabling it.
389
389
datafusion.execution.keep_partition_by_columns false Should DataFusion keep the columns used for partition_by in the output RecordBatches
390
390
datafusion.execution.listing_table_factory_infer_partitions true Should a `ListingTable` created through the `ListingTableFactory` infer table partitions from Hive compliant directories. Defaults to true (partition columns are inferred and will be represented in the table schema).
Copy file name to clipboardExpand all lines: docs/source/user-guide/configs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -140,7 +140,7 @@ The following configuration settings are available:
140
140
| datafusion.execution.enable_file_stream_work_stealing | true | When `true` (the default), DataFusion's built-in file scans dynamically rebalance files across partitions at query execution time: a partition that goes idle reads files (or byte-range morsels) originally assigned to a sibling partition, which keeps all partitions busy in a single process. Executors that depend on the plan-time partition assignment — such as Ballista and datafusion-distributed, which run each partition as an isolated task and never poll the siblings — should set this to `false` so each partition reads only its own file group and no runtime reassignment occurs. |
141
141
| datafusion.execution.skip_partial_aggregation_probe_ratio_threshold | 0.8 | Aggregation ratio (number of distinct groups / number of input rows) threshold for skipping partial aggregation. If the value is greater then partial aggregation will skip aggregation for further input |
142
142
| datafusion.execution.skip_partial_aggregation_probe_rows_threshold | 100000 | Number of input rows partial aggregation partition should process, before aggregation ratio check and trying to switch to skipping aggregation mode |
143
-
| datafusion.execution.hash_aggregate_bucket_threshold | 0 | (experimental) Number of groups above which a hash aggregation stops growing a single hash table, so that its tables stay small enough to be cache friendly. A partial aggregation then emits the state of its table and starts over, as long as the emitted groups do not come back. A final aggregation splits the groups seen so far and all further input into hash buckets, which are aggregated one after another and can be spilled and released independently. Set to 0 to disable. |
143
+
| datafusion.execution.hash_aggregate_bucket_threshold | 0 | (experimental) Number of groups above which a hash aggregation stops growing a single hash table, so that its tables stay small enough to be cache friendly. A partial aggregation then emits the state of its table and starts over, as long as the emitted groups do not come back. A final aggregation splits the groups seen so far and all further input into hash buckets, which are aggregated one after another and can be spilled and released independently; it does so at a quarter of this number when its input holds about one row per group. Aggregations of millions of groups per partition run faster and with less memory. Moving rows into buckets has a cost of its own, so aggregations that end at a few times this number of groups, string keys in particular, can run a few percent slower, and input that repeats its groups can use more memory. Set to 0 to disable. |
144
144
| datafusion.execution.use_row_number_estimates_to_optimize_partitioning | false | Should DataFusion use row number estimates at the input to decide whether increasing parallelism is beneficial or not. By default, only exact row numbers (not estimates) are used for this decision. Setting this flag to `true` will likely produce better plans. if the source of statistics is accurate. We plan to make this the default in the future. |
145
145
| datafusion.execution.enforce_batch_size_in_joins | false | Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower. Note: this option currently only applies to the symmetric hash join. |
146
146
| datafusion.execution.objectstore_writer_buffer_size | 10485760 | Size (bytes) of data buffer DataFusion uses when writing output files. This affects the size of the data chunks that are uploaded to remote object stores (e.g. AWS S3). If very large (>= 100 GiB) output files are being written, it may be necessary to increase this size to avoid errors from the remote end point. |
0 commit comments