Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
review: improve wording
  • Loading branch information
2010YOUY01 committed Oct 25, 2026
commit 8293d23e4ed886bef451f8612d3b715b8e89c946
6 changes: 6 additions & 0 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -984,6 +984,12 @@ config_namespace! {
/// the refactored implementation. When set to `false`, these joins use
/// the existing implementation. Other join types always use the existing
/// implementation.
///
/// The fallback exists only as a workaround for bugs in the new
/// implementation and will be removed, together with this option, once
/// the migration is complete.
///
/// See <https://github.com/apache/datafusion/issues/26361> for details.
pub enable_migration_hash_join: bool, default = true

/// Sets the compression codec used when spilling data to disk.
Expand Down
4 changes: 2 additions & 2 deletions datafusion/physical-plan/src/joins/hash_join/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,15 +88,15 @@ impl JoinBuildState {

/// Collecting the build side data of hash join.
///
/// It handles coordination/syncrhonization among all parallel paartitions for:
/// It handles coordination/synchronization among all parallel partitions for:
/// - Input batches collecting
/// - Dynamic filter preparation
pub(super) struct HashJoinBuild {
partition: usize,
mode: PartitionMode,

// ---------------------------------------------------
// Shared structures for coordiation among partitions.
// Shared structures for coordination among partitions.
// ---------------------------------------------------
/// Coordination for dynamic filter
build_accumulator: Option<Arc<SharedBuildAccumulator>>,
Expand Down
4 changes: 2 additions & 2 deletions datafusion/physical-plan/src/joins/hash_join/regular_join.rs
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,8 @@ impl RegularHashJoinStream {
/// see [`HashJoinBuild`] for different physical strategies
/// - Probing: Joining probe batch with build data incrementally, until probe
/// side input is exhausted.
/// - Outputting: Actually joining the probe batch and buffered build side, and
/// produce final output.
/// - Outputting: Drain buffered output, then emit unmatched build rows for
/// left/full joins.
///
/// # Transition edges
///
Expand Down
2 changes: 1 addition & 1 deletion datafusion/sqllogictest/test_files/information_schema.slt
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ datafusion.execution.collect_statistics true Should DataFusion collect statistic
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), DataFusion's built-in file scans dynamically rebalance files across partitions at query execution time: a partition that goes idle reads files (or byte-range morsels) originally assigned to a sibling partition, which keeps all partitions busy in a single process. Executors that depend on the plan-time partition assignment — such as Ballista and datafusion-distributed, which run each partition as an isolated task and never poll the siblings — should set this to `false` so each partition reads only its own file group and no runtime reassignment occurs.
datafusion.execution.enable_migration_aggregate true Whether aggregation uses the implementation from the major refactor completed in the 56.0.0 release. When set to `false`, aggregation falls back to the implementation used before 55.0.0. The fallback exists only as a workaround for bugs in the new implementation and will be removed, together with this option, after the 56.0.0 release. See <https://github.com/apache/datafusion/issues/22710> for details.
datafusion.execution.enable_migration_hash_join true Whether inner, left outer, right outer, and full outer hash joins use the refactored implementation. When set to `false`, these joins use the existing implementation. Other join types always use the existing implementation.
datafusion.execution.enable_migration_hash_join true Whether inner, left outer, right outer, and full outer hash joins use the refactored implementation. When set to `false`, these joins use the existing implementation. Other join types always use the existing implementation. The fallback exists only as a workaround for bugs in the new implementation and will be removed, together with this option, once the migration is complete. See <https://github.com/apache/datafusion/issues/26361> for details.
datafusion.execution.enable_nlj_coordinated_fallback true Enables the memory-limited fallback for `NestedLoopJoinExec` join types that emit unmatched left rows in the final output (LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK, FULL) when the right side has multiple partitions. This fallback shares left-side state (the current left chunk, the visited bitmap and the probe-thread counter) across all right-side partitions, which assumes every partition runs in the same process. Distributed engines that execute each output partition as an independent task (e.g. Ballista, datafusion-distributed) give each task its own copy of this state and poll only one partition, so the cross-partition counter never reaches zero and the fallback would stall. Such engines should set this to `false`: the coordinated fallback is then disabled for left-emitting multi-partition joins, which instead fail with a resource-exhaustion error under memory pressure rather than deadlocking. Single-partition and non-left-emitting joins are unaffected and always keep the fallback.
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower. Note: this option currently only applies to the symmetric hash join.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ The following configuration settings are available:
| datafusion.execution.planning_concurrency | 0 | Fan-out during initial physical planning. This is mostly use to plan `UNION` children in parallel. Defaults to the number of CPU cores on the system |
| datafusion.execution.skip_physical_aggregate_schema_check | false | When set to true, skips verifying that the schema produced by planning the input of `LogicalPlan::Aggregate` exactly matches the schema of the input plan. When set to false, if the schema does not match exactly (including nullability and metadata), a planning error will be raised. This is used to workaround bugs in the planner that are now caught by the new schema verification step. |
| datafusion.execution.enable_migration_aggregate | true | Whether aggregation uses the implementation from the major refactor completed in the 56.0.0 release. When set to `false`, aggregation falls back to the implementation used before 55.0.0. The fallback exists only as a workaround for bugs in the new implementation and will be removed, together with this option, after the 56.0.0 release. See <https://github.com/apache/datafusion/issues/22710> for details. |
| datafusion.execution.enable_migration_hash_join | true | Whether inner, left outer, right outer, and full outer hash joins use the refactored implementation. When set to `false`, these joins use the existing implementation. Other join types always use the existing implementation. |
| datafusion.execution.enable_migration_hash_join | true | Whether inner, left outer, right outer, and full outer hash joins use the refactored implementation. When set to `false`, these joins use the existing implementation. Other join types always use the existing implementation. The fallback exists only as a workaround for bugs in the new implementation and will be removed, together with this option, once the migration is complete. See <https://github.com/apache/datafusion/issues/26361> for details. |
| datafusion.execution.spill_compression | uncompressed | Sets the compression codec used when spilling data to disk. Since datafusion writes spill files using the Arrow IPC Stream format, only codecs supported by the Arrow IPC Stream Writer are allowed. Valid values are: uncompressed, lz4_frame, zstd. Note: lz4_frame offers faster (de)compression, but typically results in larger spill files. In contrast, zstd achieves higher compression ratios at the cost of slower (de)compression speed. |
| datafusion.execution.sort_spill_reservation_bytes | 10485760 | Specifies the reserved memory for each spillable sort operation to facilitate an in-memory merge. When a sort operation spills to disk, the in-memory data must be sorted and merged before being written to a file. This setting reserves a specific amount of memory for that in-memory sort/merge process. Note: This setting is irrelevant if the sort operation cannot spill (i.e., if there's no `DiskManager` configured). |
| datafusion.execution.sort_in_place_threshold_bytes | 1048576 | When sorting, below what size should data be concatenated and sorted in a single RecordBatch rather than sorted in batches and merged. |
Expand Down