You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 482b04b
Browse filesBrowse the repository at this point in the historyBrowse files
FileStream sibling work-stealing (WorkSource::Shared) seeds one shared work
queue from every file group and lets whichever output partition goes idle
first steal the next file or byte-range morsel. This assumes all output
partitions of a scan are polled concurrently in one process. Executors that
run each output partition as an isolated task in a separate process (Ballista,
datafusion-distributed) never poll the sibling partitions, so the single polled
partition drains the whole queue and reads files belonging to other partitions,
inflating the scan output by the partition count. That is a correctness bug for
those executors, and the existing escape hatches (preserve_order,
partitioned_by_file_group) are plan-level flags, not a session config they can
set centrally.
Add datafusion.execution.enable_file_stream_work_stealing (default true),
checked in FileScanConfig::create_sibling_state: when false it returns None so
each partition falls back to reading only its own file group. This mirrors the
enable_dynamic_filter_pushdown precedent and round-trips through datafusion-proto
as a config value. Thread ConfigOptions into DataSource::create_sibling_state so
the flag is read from the session config at execute time.
Turn the #[ignore]'d reproduction test into a passing regression test that
drives only partition 0 and checks both behaviors: with the default (stealing
on) partition 0 also reads partition 1's file, and with the flag off it reads
only its own.
Closesapache#23293.
@@ -375,6 +376,7 @@ datafusion.execution.batch_size 8192 Default batch size while creating new batch
375
376
datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
376
377
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
377
378
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
379
+
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), sibling partition streams of a single file scan share a work queue: whichever output partition goes idle first steals the next unopened file (or byte-range morsel) from the shared queue. This balances work when all output partitions are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process (for example Ballista and datafusion-distributed) never poll the sibling partitions, so the single polled partition drains the whole queue and reads files belonging to other partitions, inflating the scan output by the partition count. Such executors should set this to `false`, which makes each partition read only its own file group.
378
380
datafusion.execution.enable_migration_aggregate true Temporary switch for aggregate stream implementations that are being migrated from `GroupedHashAggregateStream`. When set to true, DataFusion tries the migrated implementations when their preconditions are satisfied. When set to false, grouped aggregation falls back to `GroupedHashAggregateStream`. This option will be removed after the migration is finished. See <https://github.com/apache/datafusion/issues/22710> for details.
379
381
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
380
382
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.
0 commit comments