Skip to content

Commit a1f64e9

Browse files
authored
fix: reject max_buffered_batches_per_output_file values below 2 (apache#24204)
## Which issue does this PR close? - Part of apache#17498 ## Rationale for this change ``` DataFusion CLI v54.1.0 > set datafusion.execution.max_buffered_batches_per_output_file = 1; > COPY (SELECT 1 as a) TO '/tmp/x.parquet'; thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:287:30: mpsc bounded channel requires buffer > 0 ``` Two call sites (`demux.rs::create_new_file_stream`, `orchestration.rs::spawn_writer_tasks_and_join`) divide `max_buffered_batches_per_output_file` in half to size a bounded `mpsc` channel's capacity. Integer division rounds both 0 *and* 1 down to 0, and Tokio's `mpsc::channel` panics on a zero capacity. A plain non-zero check (like the existing `ConfigNonZeroUsize`) would not have been sufficient here, since 1 also triggers the panic. A third call site (`demux.rs::hive_style_partitions_demuxer`) uses the raw value directly without dividing, so it would panic on 0 alone. ## What changes are included in this PR? Adds `ConfigMinTwoUsize`, mirroring the existing `ConfigNonZeroUsize` pattern already used for sibling fields (`batch_size`, `meta_fetch_concurrency`, `minimum_parallel_output_files`, etc.), and applies it to `max_buffered_batches_per_output_file`. Invalid values are now rejected with a clear configuration error at set-time instead of panicking later at write time. Updated the three read sites to call `.get()`, updated the field doc comment to explain the constraint, and regenerated `docs/source/user-guide/configs.md` via `dev/update_config_docs.sh`. ## Are these changes tested? Yes. Added two `statement error` cases to `datafusion/sqllogictest/test_files/set_variable.slt` (values 0 and 1), following the exact pattern already used for the sibling `ConfigNonZeroUsize` fields in that file. Verified manually with `datafusion-cli` that 0 and 1 now return a clean error instead of panicking, and that 2 (the default) and 3 still work correctly. Ran `cargo test -p datafusion-datasource --lib` (178 passed), the `set_variable.slt` sqllogictest suite, and `cargo check --workspace --all-targets` — all clean. ## Are there any user-facing changes? Yes: setting `datafusion.execution.max_buffered_batches_per_output_file` to `0` or `1` now returns a configuration error instead of panicking. No change for any value `>= 2` (including the default of 2).
1 parent a05388e commit a1f64e9

6 files changed

Lines changed: 120 additions & 7 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 95 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -663,6 +663,91 @@ impl Display for ConfigNonZeroUsize {
663663
}
664664
}
665665

666+
/// A `usize` configuration value that rejects 0 and 1 when set from strings.
667+
///
668+
/// Use this for options whose consumer divides the value in half to size an
669+
/// internal buffer (e.g. a bounded channel capacity): values below 2 would
670+
/// round down to a zero-capacity buffer and panic. Invalid values return a
671+
/// configuration error through [`ConfigField`] instead.
672+
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
673+
pub struct ConfigMinTwoUsize(usize);
674+
675+
/// Private helper for hard-coded defaults in `config_namespace!`, which cannot
676+
/// use `?`. All external construction should use [`ConfigMinTwoUsize::try_new`].
677+
const fn min_two_usize_default(value: usize) -> ConfigMinTwoUsize {
678+
if value >= 2 {
679+
ConfigMinTwoUsize(value)
680+
} else {
681+
panic!("value must be at least 2")
682+
}
683+
}
684+
685+
impl ConfigMinTwoUsize {
686+
/// Creates a [`ConfigMinTwoUsize`], returning a configuration error if
687+
/// `value` is less than 2.
688+
pub fn try_new(value: usize) -> Result<Self> {
689+
if value >= 2 {
690+
Ok(Self(value))
691+
} else {
692+
_config_err!("value must be at least 2")
693+
}
694+
}
695+
696+
/// Returns the wrapped `usize`.
697+
pub const fn get(self) -> usize {
698+
self.0
699+
}
700+
}
701+
702+
impl From<ConfigMinTwoUsize> for usize {
703+
fn from(value: ConfigMinTwoUsize) -> Self {
704+
value.get()
705+
}
706+
}
707+
708+
impl FromStr for ConfigMinTwoUsize {
709+
type Err = DataFusionError;
710+
711+
fn from_str(s: &str) -> Result<Self, Self::Err> {
712+
Self::try_new(default_config_transform(s)?)
713+
}
714+
}
715+
716+
impl ConfigField for ConfigMinTwoUsize {
717+
fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
718+
v.some(key, self, description)
719+
}
720+
721+
fn set(&mut self, key: &str, value: &str) -> Result<()> {
722+
if !key.is_empty() {
723+
return _config_err!(
724+
"Config field max_buffered_batches_per_output_file is a scalar ConfigMinTwoUsize and does not have nested field \"{}\"",
725+
key
726+
);
727+
}
728+
729+
*self = ConfigMinTwoUsize::from_str(value)?;
730+
Ok(())
731+
}
732+
733+
fn reset(&mut self, key: &str) -> Result<()> {
734+
if key.is_empty() {
735+
Ok(())
736+
} else {
737+
_config_err!(
738+
"Config field max_buffered_batches_per_output_file is a scalar ConfigMinTwoUsize and does not have nested field \"{}\"",
739+
key
740+
)
741+
}
742+
}
743+
}
744+
745+
impl Display for ConfigMinTwoUsize {
746+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
747+
write!(f, "{}", self.get())
748+
}
749+
}
750+
666751
/// Policy for handling duplicate keys in Spark-compatible map-construction
667752
/// functions (`map_from_arrays`, `map_from_entries`, `str_to_map`). Mirrors
668753
/// Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961).
@@ -878,8 +963,16 @@ config_namespace! {
878963
/// This is the maximum number of RecordBatches buffered
879964
/// for each output file being worked. Higher values can potentially
880965
/// give faster write performance at the cost of higher peak
881-
/// memory consumption
882-
pub max_buffered_batches_per_output_file: usize, default = 2
966+
/// memory consumption.
967+
///
968+
/// This budget is split evenly between two independent points in the
969+
/// write pipeline (see the demuxer diagram in #7791): how many files
970+
/// can be in flight from the demuxer to a writer task, and how many
971+
/// RecordBatches are buffered for a single file's writer. Must be at
972+
/// least 2 so each half gets at least 1 unit of buffering - 0 or 1
973+
/// would leave one side with a zero-capacity channel and panic at
974+
/// write time.
975+
pub max_buffered_batches_per_output_file: ConfigMinTwoUsize, default = min_two_usize_default(2)
883976

884977
/// Should sub directories be ignored when scanning directories for data
885978
/// files. Defaults to true (ignores subdirectories), consistent with

‎datafusion/datasource/src/write/demux.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ async fn row_count_demuxer(
154154
let exec_options = &context.session_config().options().execution;
155155

156156
let max_rows_per_file = exec_options.soft_max_rows_per_output_file.get();
157-
let max_buffered_batches = exec_options.max_buffered_batches_per_output_file;
157+
let max_buffered_batches = exec_options.max_buffered_batches_per_output_file.get();
158158
let minimum_parallel_files = exec_options.minimum_parallel_output_files.get();
159159
let mut part_idx = 0;
160160
let write_id = rand::distr::Alphanumeric.sample_string(&mut rand::rng(), 16);
@@ -305,7 +305,8 @@ async fn hive_style_partitions_demuxer(
305305
let write_id = rand::distr::Alphanumeric.sample_string(&mut rand::rng(), 16);
306306

307307
let exec_options = &context.session_config().options().execution;
308-
let max_buffered_recordbatches = exec_options.max_buffered_batches_per_output_file;
308+
let max_buffered_recordbatches =
309+
exec_options.max_buffered_batches_per_output_file.get();
309310

310311
// To support non string partition col types, cast the type to &str first
311312
let mut value_map: HashMap<Vec<String>, Sender<RecordBatch>> = HashMap::new();

‎datafusion/datasource/src/write/orchestration.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,7 @@ pub async fn spawn_writer_tasks_and_join(
259259
.execution
260260
.max_buffered_batches_per_output_file;
261261

262-
let (tx_file_bundle, rx_file_bundle) = mpsc::channel(rb_buffer_size / 2);
262+
let (tx_file_bundle, rx_file_bundle) = mpsc::channel(rb_buffer_size.get() / 2);
263263
let (tx_row_cnt, rx_row_cnt) = tokio::sync::oneshot::channel();
264264
let write_coordinator_task = SpawnedTask::spawn(async move {
265265
stateless_serialize_and_write_files(rx_file_bundle, tx_row_cnt).await

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -386,7 +386,7 @@ datafusion.execution.hash_join_buffering_capacity 0 How many bytes to buffer in
386386
datafusion.execution.keep_partition_by_columns false Should DataFusion keep the columns used for partition_by in the output RecordBatches
387387
datafusion.execution.listing_table_factory_infer_partitions true Should a `ListingTable` created through the `ListingTableFactory` infer table partitions from Hive compliant directories. Defaults to true (partition columns are inferred and will be represented in the table schema).
388388
datafusion.execution.listing_table_ignore_subdirectory true Should sub directories be ignored when scanning directories for data files. Defaults to true (ignores subdirectories), consistent with Hive. Note that this setting does not affect reading partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`).
389-
datafusion.execution.max_buffered_batches_per_output_file 2 This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption
389+
datafusion.execution.max_buffered_batches_per_output_file 2 This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption. This budget is split evenly between two independent points in the write pipeline (see the demuxer diagram in #7791): how many files can be in flight from the demuxer to a writer task, and how many RecordBatches are buffered for a single file's writer. Must be at least 2 so each half gets at least 1 unit of buffering - 0 or 1 would leave one side with a zero-capacity channel and panic at write time.
390390
datafusion.execution.max_spill_file_size_bytes 134217728 Maximum size in bytes for individual spill files before rotating to a new file. When operators spill data to disk (e.g., RepartitionExec), they write multiple batches to the same file until this size limit is reached, then rotate to a new file. This reduces syscall overhead compared to one-file-per-batch while preventing files from growing too large. A larger value reduces file creation overhead but may hold more disk space. A smaller value creates more files but allows finer-grained space reclamation as files can be deleted once fully consumed. Now only `RepartitionExec` supports this spill file rotation feature, other spilling operators may create spill files larger than the limit. Default: 128 MB
391391
datafusion.execution.meta_fetch_concurrency 32 Number of files to read in parallel when inferring schema and statistics
392392
datafusion.execution.minimum_parallel_output_files 4 Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached.

‎datafusion/sqllogictest/test_files/set_variable.slt‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -759,6 +759,25 @@ caused by
759759
Invalid or Unsupported Configuration: value must be greater than 0
760760

761761

762+
# max_buffered_batches_per_output_file is halved to size an internal channel
763+
# capacity, so 0 and 1 both round down to a zero-capacity channel and must be
764+
# rejected, not just 0.
765+
statement error
766+
SET datafusion.execution.max_buffered_batches_per_output_file = 0
767+
----
768+
DataFusion error: Error setting config datafusion.execution.max_buffered_batches_per_output_file
769+
caused by
770+
Invalid or Unsupported Configuration: value must be at least 2
771+
772+
773+
statement error
774+
SET datafusion.execution.max_buffered_batches_per_output_file = 1
775+
----
776+
DataFusion error: Error setting config datafusion.execution.max_buffered_batches_per_output_file
777+
caused by
778+
Invalid or Unsupported Configuration: value must be at least 2
779+
780+
762781
# Config reset
763782
statement ok
764783
RESET datafusion.catalog.create_default_catalog_and_schema

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,7 @@ The following configuration settings are available:
130130
| datafusion.execution.meta_fetch_concurrency | 32 | Number of files to read in parallel when inferring schema and statistics |
131131
| datafusion.execution.minimum_parallel_output_files | 4 | Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached. |
132132
| datafusion.execution.soft_max_rows_per_output_file | 50000000 | Target number of rows in output files when writing multiple. This is a soft max, so it can be exceeded slightly. There also will be one file smaller than the limit if the total number of rows written is not roughly divisible by the soft max |
133-
| datafusion.execution.max_buffered_batches_per_output_file | 2 | This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption |
133+
| datafusion.execution.max_buffered_batches_per_output_file | 2 | This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption. This budget is split evenly between two independent points in the write pipeline (see the demuxer diagram in #7791): how many files can be in flight from the demuxer to a writer task, and how many RecordBatches are buffered for a single file's writer. Must be at least 2 so each half gets at least 1 unit of buffering - 0 or 1 would leave one side with a zero-capacity channel and panic at write time. |
134134
| datafusion.execution.listing_table_ignore_subdirectory | true | Should sub directories be ignored when scanning directories for data files. Defaults to true (ignores subdirectories), consistent with Hive. Note that this setting does not affect reading partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`). |
135135
| datafusion.execution.listing_table_factory_infer_partitions | true | Should a `ListingTable` created through the `ListingTableFactory` infer table partitions from Hive compliant directories. Defaults to true (partition columns are inferred and will be represented in the table schema). |
136136
| datafusion.execution.enable_recursive_ctes | true | Should DataFusion support recursive CTEs |

0 commit comments

Comments
 (0)