Skip to content

Commit 482b04b

Browse files
committed
feat: add datafusion.execution.enable_file_stream_work_stealing config
FileStream sibling work-stealing (WorkSource::Shared) seeds one shared work queue from every file group and lets whichever output partition goes idle first steal the next file or byte-range morsel. This assumes all output partitions of a scan are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process (Ballista, datafusion-distributed) never poll the sibling partitions, so the single polled partition drains the whole queue and reads files belonging to other partitions, inflating the scan output by the partition count. That is a correctness bug for those executors, and the existing escape hatches (preserve_order, partitioned_by_file_group) are plan-level flags, not a session config they can set centrally. Add datafusion.execution.enable_file_stream_work_stealing (default true), checked in FileScanConfig::create_sibling_state: when false it returns None so each partition falls back to reading only its own file group. This mirrors the enable_dynamic_filter_pushdown precedent and round-trips through datafusion-proto as a config value. Thread ConfigOptions into DataSource::create_sibling_state so the flag is read from the session config at execute time. Turn the #[ignore]'d reproduction test into a passing regression test that drives only partition 0 and checks both behaviors: with the default (stealing on) partition 0 also reads partition 1's file, and with the flag off it reads only its own. Closes apache#23293.
1 parent 1b2760b commit 482b04b

6 files changed

Lines changed: 114 additions & 59 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -903,6 +903,21 @@ config_namespace! {
903903
/// Should DataFusion keep the columns used for partition_by in the output RecordBatches
904904
pub keep_partition_by_columns: bool, default = false
905905

906+
/// When `true` (the default), sibling partition streams of a single file
907+
/// scan share a work queue: whichever output partition goes idle first
908+
/// steals the next unopened file (or byte-range morsel) from the shared
909+
/// queue. This balances work when all output partitions are polled
910+
/// concurrently in one process.
911+
///
912+
/// Executors that run each output partition as an isolated task in a
913+
/// separate process (for example Ballista and datafusion-distributed)
914+
/// never poll the sibling partitions, so the single polled partition
915+
/// drains the whole queue and reads files belonging to other partitions,
916+
/// inflating the scan output by the partition count. Such executors
917+
/// should set this to `false`, which makes each partition read only its
918+
/// own file group.
919+
pub enable_file_stream_work_stealing: bool, default = true
920+
906921
/// Aggregation ratio (number of distinct groups / number of input rows)
907922
/// threshold for skipping partial aggregation. If the value is greater
908923
/// then partial aggregation will skip aggregation for further input

‎datafusion/datasource/src/file_scan_config/mod.rs‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,12 +1124,18 @@ impl DataSource for FileScanConfig {
11241124
/// during one execution.
11251125
///
11261126
/// This returns `None` when sibling streams must not share work, such as
1127-
/// when file order must be preserved or the file groups define the output
1128-
/// partitioning needed for the rest of the plan
1129-
fn create_sibling_state(&self) -> Option<Arc<dyn Any + Send + Sync>> {
1127+
/// when file order must be preserved, the file groups define the output
1128+
/// partitioning needed for the rest of the plan, or work stealing is
1129+
/// disabled via
1130+
/// `datafusion.execution.enable_file_stream_work_stealing`.
1131+
fn create_sibling_state(
1132+
&self,
1133+
config: &ConfigOptions,
1134+
) -> Option<Arc<dyn Any + Send + Sync>> {
11301135
if self.preserve_order
11311136
|| self.output_partitioning.is_some()
11321137
|| self.partitioned_by_file_group
1138+
|| !config.execution.enable_file_stream_work_stealing
11331139
{
11341140
return None;
11351141
}

‎datafusion/datasource/src/file_stream/mod.rs‎

Lines changed: 79 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,7 @@ mod tests {
182182
use arrow::array::{AsArray, RecordBatch};
183183
use arrow::datatypes::{DataType, Field, Int32Type, Schema};
184184
use datafusion_common::DataFusionError;
185+
use datafusion_common::config::ConfigOptions;
185186
use datafusion_common::error::Result;
186187
use datafusion_execution::object_store::ObjectStoreUrl;
187188
use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
@@ -1131,72 +1132,94 @@ mod tests {
11311132
Ok(())
11321133
}
11331134

1134-
/// Reproduces <https://github.com/apache/datafusion/issues/23293>.
1135+
/// Covers <https://github.com/apache/datafusion/issues/23293>.
11351136
///
11361137
/// Executors that run each output partition as an isolated task in a
11371138
/// separate process (Ballista, datafusion-distributed) build one plan
11381139
/// instance and call `execute(partition)` for a single partition. The
11391140
/// sibling partitions are never polled in that process. A `DataSourceExec`
11401141
/// seeds one shared work queue from every file group, so the single polled
11411142
/// partition drains the whole queue and reads files that belong to other
1142-
/// partitions. Each isolated task does the same, inflating the scan output
1143-
/// by the partition count.
1143+
/// partitions, inflating the scan output by the partition count.
11441144
///
1145-
/// This test models that setup: it builds and drives only partition 0.
1146-
/// Partition 0 should read only its own file (`file1.parquet`), but today
1147-
/// it also reads partition 1's file, so it fails with
1148-
/// `"Batch: 101\nBatch: 201"`.
1149-
///
1150-
/// Ignored because it fails on `main` by design: it is a caught regression
1151-
/// kept with its assertion intact. Run it with
1152-
/// `cargo test -p datafusion-datasource -- --ignored
1153-
/// regression_isolated_partition_reads_only_its_own_files`.
1145+
/// `datafusion.execution.enable_file_stream_work_stealing` turns the shared
1146+
/// queue off: `create_sibling_state` then returns `None` and each partition
1147+
/// reads only its own file group. This test drives only partition 0 (as an
1148+
/// isolated task would) and checks both behaviors: with the default
1149+
/// (stealing on) partition 0 also reads partition 1's file, and with the
1150+
/// flag off it reads only its own.
11541151
#[tokio::test]
1155-
#[ignore = "reproduces #23293: shared FileStream work queue is drained by a single isolated task"]
1156-
async fn regression_isolated_partition_reads_only_its_own_files() -> Result<()> {
1157-
let test = FileStreamMorselTest::new()
1158-
.with_file_in_partition(
1159-
PartitionId(0),
1160-
MockPlanner::builder("file1.parquet")
1161-
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(10), 101))
1162-
.return_none(),
1163-
)
1164-
.with_file_in_partition(
1165-
PartitionId(1),
1166-
MockPlanner::builder("file2.parquet")
1167-
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(11), 201))
1168-
.return_none(),
1169-
);
1152+
async fn isolated_partition_respects_work_stealing_config() -> Result<()> {
1153+
// A two-file scan: partition 0 owns file1 (batch 101), partition 1 owns
1154+
// file2 (batch 201). Rebuilt per phase because the mock planners are
1155+
// consumed as files are opened.
1156+
let make_test = || {
1157+
FileStreamMorselTest::new()
1158+
.with_file_in_partition(
1159+
PartitionId(0),
1160+
MockPlanner::builder("file1.parquet")
1161+
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(10), 101))
1162+
.return_none(),
1163+
)
1164+
.with_file_in_partition(
1165+
PartitionId(1),
1166+
MockPlanner::builder("file2.parquet")
1167+
.add_plan(MockPlanBuilder::new().with_morsel(MorselId(11), 201))
1168+
.return_none(),
1169+
)
1170+
};
11701171

1172+
// Drive only partition 0, exactly as an isolated per-task executor does.
1173+
// Partition 1's stream is never created.
1174+
async fn drive_partition0(
1175+
test: &FileStreamMorselTest,
1176+
config: &FileScanConfig,
1177+
options: &ConfigOptions,
1178+
) -> Result<(bool, String)> {
1179+
let shared_work_source =
1180+
config.create_sibling_state(options).and_then(|state| {
1181+
state.as_ref().downcast_ref::<SharedWorkSource>().cloned()
1182+
});
1183+
let shared = shared_work_source.is_some();
1184+
let metrics_set = ExecutionPlanMetricsSet::new();
1185+
let partition0 = FileStreamBuilder::new(config)
1186+
.with_partition(0)
1187+
.with_shared_work_source(shared_work_source)
1188+
.with_morselizer(Box::new(test.morselizer.clone()))
1189+
.with_metrics(&metrics_set)
1190+
.build()?;
1191+
Ok((shared, drain_stream_output(partition0).await?))
1192+
}
1193+
1194+
// With work stealing enabled (the default), the single polled partition
1195+
// drains the shared queue and reads a sibling's file (issue #23293).
1196+
let mut options = ConfigOptions::default();
1197+
assert!(options.execution.enable_file_stream_work_stealing);
1198+
let test = make_test();
11711199
let config = test.test_config();
1172-
let metrics_set = ExecutionPlanMetricsSet::new();
1173-
1174-
// A `DataSourceExec` builds exactly one shared work source per
1175-
// execution, seeded from every file group. An isolated task builds that
1176-
// state and then executes only its own partition.
1177-
let shared_work_source = config
1178-
.create_sibling_state()
1179-
.and_then(|state| state.as_ref().downcast_ref::<SharedWorkSource>().cloned());
1200+
let (shared, output) = drive_partition0(&test, &config, &options).await?;
11801201
assert!(
1181-
shared_work_source.is_some(),
1182-
"a plain repartitioned scan should use a shared work source"
1202+
shared,
1203+
"the default config shares one work queue across siblings"
1204+
);
1205+
assert_eq!(
1206+
output, "Batch: 101\nBatch: 201",
1207+
"with work stealing on, an isolated partition drains the shared queue"
11831208
);
11841209

1185-
// Build and drive ONLY partition 0. Partition 1's stream is never
1186-
// created, exactly as in an isolated per-task executor.
1187-
let partition0 = FileStreamBuilder::new(&config)
1188-
.with_partition(0)
1189-
.with_shared_work_source(shared_work_source)
1190-
.with_morselizer(Box::new(test.morselizer.clone()))
1191-
.with_metrics(&metrics_set)
1192-
.build()?;
1193-
1194-
let output = drain_stream_output(partition0).await?;
1195-
1210+
// Disabling work stealing drops the shared queue, so partition 0 reads
1211+
// only its own file group.
1212+
options.execution.enable_file_stream_work_stealing = false;
1213+
let test = make_test();
1214+
let config = test.test_config();
1215+
let (shared, output) = drive_partition0(&test, &config, &options).await?;
1216+
assert!(
1217+
!shared,
1218+
"disabling work stealing drops the shared work source"
1219+
);
11961220
assert_eq!(
11971221
output, "Batch: 101",
1198-
"partition 0 read files belonging to a sibling partition: a single \
1199-
isolated task drained the shared work queue (issue #23293)"
1222+
"with work stealing off, partition 0 reads only its own file group"
12001223
);
12011224

12021225
Ok(())
@@ -1287,7 +1310,7 @@ mod tests {
12871310
let unlimited_config = test.test_config();
12881311
let limited_config = test.clone().with_limit(1).test_config();
12891312
let shared_work_source = limited_config
1290-
.create_sibling_state()
1313+
.create_sibling_state(&ConfigOptions::default())
12911314
.and_then(|state| state.as_ref().downcast_ref::<SharedWorkSource>().cloned())
12921315
.expect("shared work source");
12931316
let limited_metrics = ExecutionPlanMetricsSet::new();
@@ -1539,9 +1562,11 @@ mod tests {
15391562
// `FileStream`s directly, bypassing `DataSourceExec`, so they must
15401563
// perform the same setup explicitly when exercising sibling-stream
15411564
// work stealing.
1542-
let shared_work_source = config.create_sibling_state().and_then(|state| {
1543-
state.as_ref().downcast_ref::<SharedWorkSource>().cloned()
1544-
});
1565+
let shared_work_source = config
1566+
.create_sibling_state(&ConfigOptions::default())
1567+
.and_then(|state| {
1568+
state.as_ref().downcast_ref::<SharedWorkSource>().cloned()
1569+
});
15451570
if !self.build_streams_on_first_read {
15461571
for partition in build_order {
15471572
let stream = FileStreamBuilder::new(&config)

‎datafusion/datasource/src/source.rs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,10 @@ pub trait DataSource: Any + Send + Sync + Debug {
246246
///
247247
/// Returns `None` (the default) if this data source has
248248
/// no sibling-shared execution state.
249-
fn create_sibling_state(&self) -> Option<Arc<dyn Any + Send + Sync>> {
249+
fn create_sibling_state(
250+
&self,
251+
_config: &ConfigOptions,
252+
) -> Option<Arc<dyn Any + Send + Sync>> {
250253
None
251254
}
252255

@@ -392,7 +395,10 @@ impl ExecutionPlan for DataSourceExec {
392395
) -> Result<SendableRecordBatchStream> {
393396
let shared_state = self
394397
.execution_state
395-
.get_or_init(|| self.data_source.create_sibling_state())
398+
.get_or_init(|| {
399+
self.data_source
400+
.create_sibling_state(context.session_config().options())
401+
})
396402
.clone();
397403
let args = OpenArgs::new(partition, Arc::clone(&context))
398404
.with_shared_state(shared_state);

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@ datafusion.execution.batch_size 8192
218218
datafusion.execution.coalesce_batches true
219219
datafusion.execution.collect_statistics true
220220
datafusion.execution.enable_ansi_mode false
221+
datafusion.execution.enable_file_stream_work_stealing true
221222
datafusion.execution.enable_migration_aggregate true
222223
datafusion.execution.enable_recursive_ctes true
223224
datafusion.execution.enforce_batch_size_in_joins false
@@ -375,6 +376,7 @@ datafusion.execution.batch_size 8192 Default batch size while creating new batch
375376
datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
376377
datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
377378
datafusion.execution.enable_ansi_mode false Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.
379+
datafusion.execution.enable_file_stream_work_stealing true When `true` (the default), sibling partition streams of a single file scan share a work queue: whichever output partition goes idle first steals the next unopened file (or byte-range morsel) from the shared queue. This balances work when all output partitions are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process (for example Ballista and datafusion-distributed) never poll the sibling partitions, so the single polled partition drains the whole queue and reads files belonging to other partitions, inflating the scan output by the partition count. Such executors should set this to `false`, which makes each partition read only its own file group.
378380
datafusion.execution.enable_migration_aggregate true Temporary switch for aggregate stream implementations that are being migrated from `GroupedHashAggregateStream`. When set to true, DataFusion tries the migrated implementations when their preconditions are satisfied. When set to false, grouped aggregation falls back to `GroupedHashAggregateStream`. This option will be removed after the migration is finished. See <https://github.com/apache/datafusion/issues/22710> for details.
379381
datafusion.execution.enable_recursive_ctes true Should DataFusion support recursive CTEs
380382
datafusion.execution.enforce_batch_size_in_joins false Should DataFusion enforce batch size in joins or not. By default, DataFusion will not enforce batch size in joins. Enforcing batch size in joins can reduce memory usage when joining large tables with a highly-selective join filter, but is also slightly slower.

0 commit comments

Comments
 (0)