Skip to content

Commit 608915e

Browse files
committed
refactor: address review feedback on work-stealing config
- Reword the config docstring in user-visible terms (dynamic runtime rebalancing) rather than internal work-queue mechanics. - Note on FileScanConfig::file_groups that files may be reassigned across partitions at runtime when work stealing is enabled unless preserve_order or partitioned_by_file_group is set. - Replace the bespoke isolated-partition test with the file's standard morsel snapshot harness: FileStreamMorselTest gains with_enable_file_stream_work_stealing, and the test reuses two_partition_morsel_test to show that disabling the flag keeps each partition's files local. Regenerate configs.md and information_schema.slt for the reworded docstring.
1 parent b836a6d commit 608915e

5 files changed

Lines changed: 60 additions & 102 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -903,19 +903,17 @@ config_namespace! {
903903
/// Should DataFusion keep the columns used for partition_by in the output RecordBatches
904904
pub keep_partition_by_columns: bool, default = false
905905

906-
/// When `true` (the default), sibling partition streams of a single file
907-
/// scan share a work queue: whichever output partition goes idle first
908-
/// steals the next unopened file (or byte-range morsel) from the shared
909-
/// queue. This balances work when all output partitions are polled
910-
/// concurrently in one process.
906+
/// When `true` (the default), DataFusion's built-in file scans
907+
/// dynamically rebalance files across partitions at query execution
908+
/// time: a partition that goes idle reads files (or byte-range morsels)
909+
/// originally assigned to a sibling partition, which keeps all
910+
/// partitions busy in a single process.
911911
///
912-
/// Executors that run each output partition as an isolated task in a
913-
/// separate process (for example Ballista and datafusion-distributed)
914-
/// never poll the sibling partitions, so the single polled partition
915-
/// drains the whole queue and reads files belonging to other partitions,
916-
/// inflating the scan output by the partition count. Such executors
917-
/// should set this to `false`, which makes each partition read only its
918-
/// own file group.
912+
/// Executors that depend on the plan-time partition assignment — such as
913+
/// Ballista and datafusion-distributed, which run each partition as an
914+
/// isolated task and never poll the siblings — should set this to
915+
/// `false` so each partition reads only its own file group and no
916+
/// runtime reassignment occurs.
919917
pub enable_file_stream_work_stealing: bool, default = true
920918

921919
/// Aggregation ratio (number of distinct groups / number of input rows)

‎datafusion/datasource/src/file_scan_config/mod.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,12 @@ pub struct FileScanConfig {
159159
/// DataFusion may attempt to read each partition of files
160160
/// concurrently, however files *within* a partition will be read
161161
/// sequentially, one after the next.
162+
///
163+
/// Note that when `datafusion.execution.enable_file_stream_work_stealing`
164+
/// is enabled (the default), files may be reassigned to a different
165+
/// partition at runtime unless `preserve_order` or
166+
/// `partitioned_by_file_group` is set, so a file is not guaranteed to be
167+
/// read by the partition it is grouped under here.
162168
pub file_groups: Vec<FileGroup>,
163169
/// Table constraints
164170
pub constraints: Constraints,

‎datafusion/datasource/src/file_stream/mod.rs‎

Lines changed: 42 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -1132,94 +1132,36 @@ mod tests {
11321132
Ok(())
11331133
}
11341134

1135-
/// Covers <https://github.com/apache/datafusion/issues/23293>.
1135+
/// Verifies that disabling `enable_file_stream_work_stealing` keeps each
1136+
/// stream's files local, so a sibling cannot steal them at runtime.
11361137
///
1137-
/// Executors that run each output partition as an isolated task in a
1138-
/// separate process (Ballista, datafusion-distributed) build one plan
1139-
/// instance and call `execute(partition)` for a single partition. The
1140-
/// sibling partitions are never polled in that process. A `DataSourceExec`
1141-
/// seeds one shared work queue from every file group, so the single polled
1142-
/// partition drains the whole queue and reads files that belong to other
1143-
/// partitions, inflating the scan output by the partition count.
1144-
///
1145-
/// `datafusion.execution.enable_file_stream_work_stealing` turns the shared
1146-
/// queue off: `create_sibling_state` then returns `None` and each partition
1147-
/// reads only its own file group. This test drives only partition 0 (as an
1148-
/// isolated task would) and checks both behaviors: with the default
1149-
/// (stealing on) partition 0 also reads partition 1's file, and with the
1150-
/// flag off it reads only its own.
1138+
/// Covers <https://github.com/apache/datafusion/issues/23293>: executors
1139+
/// that run each output partition as an isolated task in a separate process
1140+
/// (Ballista, datafusion-distributed) poll only their own partition, so the
1141+
/// shared work queue would let that one partition drain files belonging to
1142+
/// its siblings. Disabling the flag falls back to per-partition file groups.
11511143
#[tokio::test]
1152-
async fn isolated_partition_respects_work_stealing_config() -> Result<()> {
1153-
// A two-file scan: partition 0 owns file1 (batch 101), partition 1 owns
1154-
// file2 (batch 201). Rebuilt per phase because the mock planners are
1155-
// consumed as files are opened.
1156-
let make_test = || {
1157-
FileStreamMorselTest::new()
1158-
.with_file_in_partition(
1159-
PartitionId(0),
1160-
MockPlanner::builder("file1.parquet")
1161-
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(10), 101))
1162-
.return_none(),
1163-
)
1164-
.with_file_in_partition(
1165-
PartitionId(1),
1166-
MockPlanner::builder("file2.parquet")
1167-
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(11), 201))
1168-
.return_none(),
1169-
)
1170-
};
1171-
1172-
// Drive only partition 0, exactly as an isolated per-task executor does.
1173-
// Partition 1's stream is never created.
1174-
async fn drive_partition0(
1175-
test: &FileStreamMorselTest,
1176-
config: &FileScanConfig,
1177-
options: &ConfigOptions,
1178-
) -> Result<String> {
1179-
let shared_work_source =
1180-
config.create_sibling_state(options).and_then(|state| {
1181-
state.as_ref().downcast_ref::<SharedWorkSource>().cloned()
1182-
});
1183-
let metrics_set = ExecutionPlanMetricsSet::new();
1184-
let partition0 = FileStreamBuilder::new(config)
1185-
.with_partition(0)
1186-
.with_shared_work_source(shared_work_source)
1187-
.with_morselizer(Box::new(test.morselizer.clone()))
1188-
.with_metrics(&metrics_set)
1189-
.build()?;
1190-
drain_stream_output(partition0).await
1191-
}
1192-
1193-
// With work stealing enabled (the default), the single polled partition
1194-
// drains the shared queue and reads a sibling's file (issue #23293).
1195-
let mut options = ConfigOptions::default();
1196-
assert!(options.execution.enable_file_stream_work_stealing);
1197-
let test = make_test();
1198-
let config = test.test_config();
1199-
assert!(
1200-
config.create_sibling_state(&options).is_some(),
1201-
"the default config shares one work queue across siblings"
1202-
);
1203-
assert_eq!(
1204-
drive_partition0(&test, &config, &options).await?,
1205-
"Batch: 101\nBatch: 201",
1206-
"with work stealing on, an isolated partition drains the shared queue"
1207-
);
1144+
async fn morsel_disabled_work_stealing_keeps_files_local() -> Result<()> {
1145+
// same fixture as `morsel_shared_files_can_be_stolen`, but with work
1146+
// stealing disabled via config
1147+
let test = two_partition_morsel_test()
1148+
.with_enable_file_stream_work_stealing(false)
1149+
.with_file_stream_events(false);
12081150

1209-
// Disabling work stealing drops the shared queue, so partition 0 reads
1210-
// only its own file group.
1211-
options.execution.enable_file_stream_work_stealing = false;
1212-
let test = make_test();
1213-
let config = test.test_config();
1214-
assert!(
1215-
config.create_sibling_state(&options).is_none(),
1216-
"disabling work stealing drops the shared work source"
1217-
);
1218-
assert_eq!(
1219-
drive_partition0(&test, &config, &options).await?,
1220-
"Batch: 101",
1221-
"with work stealing off, partition 0 reads only its own file group"
1222-
);
1151+
// Even though Partition 1 is polled first, it cannot steal the three
1152+
// files assigned to Partition 0; each partition reads only its own.
1153+
insta::assert_snapshot!(test.run().await.unwrap(), @r"
1154+
----- Partition 0 -----
1155+
Batch: 101
1156+
Batch: 102
1157+
Batch: 103
1158+
Done
1159+
----- Partition 1 -----
1160+
Batch: 201
1161+
Done
1162+
----- File Stream Events -----
1163+
(omitted due to with_file_stream_events(false))
1164+
");
12231165

12241166
Ok(())
12251167
}
@@ -1425,6 +1367,7 @@ mod tests {
14251367
partition_files: BTreeMap<PartitionId, Vec<String>>,
14261368
preserve_order: bool,
14271369
partitioned_by_file_group: bool,
1370+
enable_file_stream_work_stealing: bool,
14281371
file_stream_events: bool,
14291372
build_streams_on_first_read: bool,
14301373
reads: Vec<PartitionId>,
@@ -1439,6 +1382,7 @@ mod tests {
14391382
partition_files: BTreeMap::new(),
14401383
preserve_order: false,
14411384
partitioned_by_file_group: false,
1385+
enable_file_stream_work_stealing: true,
14421386
file_stream_events: true,
14431387
build_streams_on_first_read: false,
14441388
reads: vec![],
@@ -1484,6 +1428,14 @@ mod tests {
14841428
self
14851429
}
14861430

1431+
/// Sets `datafusion.execution.enable_file_stream_work_stealing`. When
1432+
/// disabled, each stream keeps its own files local instead of sharing a
1433+
/// work queue with its siblings.
1434+
fn with_enable_file_stream_work_stealing(mut self, enable: bool) -> Self {
1435+
self.enable_file_stream_work_stealing = enable;
1436+
self
1437+
}
1438+
14871439
/// Controls whether scheduler events are included in the snapshot.
14881440
///
14891441
/// When disabled, `run()` still includes the event section header but
@@ -1561,9 +1513,11 @@ mod tests {
15611513
// `FileStream`s directly, bypassing `DataSourceExec`, so they must
15621514
// perform the same setup explicitly when exercising sibling-stream
15631515
// work stealing.
1564-
let shared_work_source = config
1565-
.create_sibling_state(&ConfigOptions::default())
1566-
.and_then(|state| {
1516+
let mut options = ConfigOptions::default();
1517+
options.execution.enable_file_stream_work_stealing =
1518+
self.enable_file_stream_work_stealing;
1519+
let shared_work_source =
1520+
config.create_sibling_state(&options).and_then(|state| {
15671521
state.as_ref().downcast_ref::<SharedWorkSource>().cloned()
15681522
});
15691523
if !self.build_streams_on_first_read {

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -376,7 +376,7 @@ datafusion.execution.batch_size 8192 Default batch size while creating new batch
376376
datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
377377
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
378378
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
379-
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), sibling partition streams of a single file scan share a work queue: whichever output partition goes idle first steals the next unopened file (or byte-range morsel) from the shared queue. This balances work when all output partitions are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process (for example Ballista and datafusion-distributed) never poll the sibling partitions, so the single polled partition drains the whole queue and reads files belonging to other partitions, inflating the scan output by the partition count. Such executors should set this to `false`, which makes each partition read only its own file group.
379+
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), DataFusion's built-in file scans dynamically rebalance files across partitions at query execution time: a partition that goes idle reads files (or byte-range morsels) originally assigned to a sibling partition, which keeps all partitions busy in a single process. Executors that depend on the plan-time partition assignment — such as Ballista and datafusion-distributed, which run each partition as an isolated task and never poll the siblings — should set this to `false` so each partition reads only its own file group and no runtime reassignment occurs.
380380
datafusion.execution.enable_migration_aggregate true Temporary switch for aggregate stream implementations that are being migrated from `GroupedHashAggregateStream`. When set to true, DataFusion tries the migrated implementations when their preconditions are satisfied. When set to false, grouped aggregation falls back to `GroupedHashAggregateStream`. This option will be removed after the migration is finished. See <https://github.com/apache/datafusion/issues/22710> for details.
381381
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
382382
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.

0 commit comments

Comments
 (0)