You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 608915e
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: address review feedback on work-stealing config
- Reword the config docstring in user-visible terms (dynamic runtime
rebalancing) rather than internal work-queue mechanics.
- Note on FileScanConfig::file_groups that files may be reassigned across
partitions at runtime when work stealing is enabled unless preserve_order or
partitioned_by_file_group is set.
- Replace the bespoke isolated-partition test with the file's standard morsel
snapshot harness: FileStreamMorselTest gains with_enable_file_stream_work_stealing,
and the test reuses two_partition_morsel_test to show that disabling the flag
keeps each partition's files local.
Regenerate configs.md and information_schema.slt for the reworded docstring.
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/information_schema.slt
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -376,7 +376,7 @@ datafusion.execution.batch_size 8192 Default batch size while creating new batch
376
376
datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
377
377
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
378
378
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
379
-
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), sibling partition streams of a single file scan share a work queue: whichever output partition goes idle first steals the next unopened file (or byte-range morsel) from the shared queue. This balances work when all output partitions are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process (for example Ballista and datafusion-distributed) never poll the sibling partitions, so the single polled partition drains the whole queue and reads files belonging to other partitions, inflating the scan output by the partition count. Such executors should set this to `false`, which makes each partition read only its own file group.
379
+
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), DataFusion's built-in file scans dynamically rebalance files across partitions at query execution time: a partition that goes idle reads files (or byte-range morsels) originally assigned to a sibling partition, which keeps all partitions busy in a single process. Executors that depend on the plan-time partition assignment — such as Ballista and datafusion-distributed, which run each partition as an isolated task and never poll the siblings — should set this to `false` so each partition reads only its own file group and no runtime reassignment occurs.
380
380
datafusion.execution.enable_migration_aggregate true Temporary switch for aggregate stream implementations that are being migrated from `GroupedHashAggregateStream`. When set to true, DataFusion tries the migrated implementations when their preconditions are satisfied. When set to false, grouped aggregation falls back to `GroupedHashAggregateStream`. This option will be removed after the migration is finished. See <https://github.com/apache/datafusion/issues/22710> for details.
381
381
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
382
382
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.
0 commit comments