You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a1f64e9
Browse filesBrowse the repository at this point in the historyBrowse files
## Which issue does this PR close?
- Part of apache#17498
## Rationale for this change
```
DataFusion CLI v54.1.0
> set datafusion.execution.max_buffered_batches_per_output_file = 1;
> COPY (SELECT 1 as a) TO '/tmp/x.parquet';
thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:287:30:
mpsc bounded channel requires buffer > 0
```
Two call sites (`demux.rs::create_new_file_stream`,
`orchestration.rs::spawn_writer_tasks_and_join`) divide
`max_buffered_batches_per_output_file` in half to size a bounded `mpsc`
channel's capacity. Integer division rounds both 0 *and* 1 down to 0,
and Tokio's `mpsc::channel` panics on a zero capacity. A plain non-zero
check (like the existing `ConfigNonZeroUsize`) would not have been
sufficient here, since 1 also triggers the panic.
A third call site (`demux.rs::hive_style_partitions_demuxer`) uses the
raw value directly without dividing, so it would panic on 0 alone.
## What changes are included in this PR?
Adds `ConfigMinTwoUsize`, mirroring the existing `ConfigNonZeroUsize`
pattern already used for sibling fields (`batch_size`,
`meta_fetch_concurrency`, `minimum_parallel_output_files`, etc.), and
applies it to `max_buffered_batches_per_output_file`. Invalid values are
now rejected with a clear configuration error at set-time instead of
panicking later at write time. Updated the three read sites to call
`.get()`, updated the field doc comment to explain the constraint, and
regenerated `docs/source/user-guide/configs.md` via
`dev/update_config_docs.sh`.
## Are these changes tested?
Yes. Added two `statement error` cases to
`datafusion/sqllogictest/test_files/set_variable.slt` (values 0 and 1),
following the exact pattern already used for the sibling
`ConfigNonZeroUsize` fields in that file. Verified manually with
`datafusion-cli` that 0 and 1 now return a clean error instead of
panicking, and that 2 (the default) and 3 still work correctly. Ran
`cargo test -p datafusion-datasource --lib` (178 passed), the
`set_variable.slt` sqllogictest suite, and `cargo check --workspace
--all-targets` — all clean.
## Are there any user-facing changes?
Yes: setting `datafusion.execution.max_buffered_batches_per_output_file`
to `0` or `1` now returns a configuration error instead of panicking. No
change for any value `>= 2` (including the default of 2).
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/information_schema.slt
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -386,7 +386,7 @@ datafusion.execution.hash_join_buffering_capacity 0 How many bytes to buffer in
386
386
datafusion.execution.keep_partition_by_columns false Should DataFusion keep the columns used for partition_by in the output RecordBatches
387
387
datafusion.execution.listing_table_factory_infer_partitions true Should a `ListingTable` created through the `ListingTableFactory` infer table partitions from Hive compliant directories. Defaults to true (partition columns are inferred and will be represented in the table schema).
388
388
datafusion.execution.listing_table_ignore_subdirectory true Should sub directories be ignored when scanning directories for data files. Defaults to true (ignores subdirectories), consistent with Hive. Note that this setting does not affect reading partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`).
389
-
datafusion.execution.max_buffered_batches_per_output_file 2 This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption
389
+
datafusion.execution.max_buffered_batches_per_output_file 2 This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption. This budget is split evenly between two independent points in the write pipeline (see the demuxer diagram in #7791): how many files can be in flight from the demuxer to a writer task, and how many RecordBatches are buffered for a single file's writer. Must be at least 2 so each half gets at least 1 unit of buffering - 0 or 1 would leave one side with a zero-capacity channel and panic at write time.
390
390
datafusion.execution.max_spill_file_size_bytes 134217728 Maximum size in bytes for individual spill files before rotating to a new file. When operators spill data to disk (e.g., RepartitionExec), they write multiple batches to the same file until this size limit is reached, then rotate to a new file. This reduces syscall overhead compared to one-file-per-batch while preventing files from growing too large. A larger value reduces file creation overhead but may hold more disk space. A smaller value creates more files but allows finer-grained space reclamation as files can be deleted once fully consumed. Now only `RepartitionExec` supports this spill file rotation feature, other spilling operators may create spill files larger than the limit. Default: 128 MB
391
391
datafusion.execution.meta_fetch_concurrency 32 Number of files to read in parallel when inferring schema and statistics
392
392
datafusion.execution.minimum_parallel_output_files 4 Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached.
Copy file name to clipboardExpand all lines: docs/source/user-guide/configs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -130,7 +130,7 @@ The following configuration settings are available:
130
130
| datafusion.execution.meta_fetch_concurrency | 32 | Number of files to read in parallel when inferring schema and statistics |
131
131
| datafusion.execution.minimum_parallel_output_files | 4 | Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached. |
132
132
| datafusion.execution.soft_max_rows_per_output_file | 50000000 | Target number of rows in output files when writing multiple. This is a soft max, so it can be exceeded slightly. There also will be one file smaller than the limit if the total number of rows written is not roughly divisible by the soft max |
133
-
| datafusion.execution.max_buffered_batches_per_output_file | 2 | This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption |
133
+
| datafusion.execution.max_buffered_batches_per_output_file | 2 | This is the maximum number of RecordBatches buffered for each output file being worked. Higher values can potentially give faster write performance at the cost of higher peak memory consumption. This budget is split evenly between two independent points in the write pipeline (see the demuxer diagram in #7791): how many files can be in flight from the demuxer to a writer task, and how many RecordBatches are buffered for a single file's writer. Must be at least 2 so each half gets at least 1 unit of buffering - 0 or 1 would leave one side with a zero-capacity channel and panic at write time. |
134
134
| datafusion.execution.listing_table_ignore_subdirectory | true | Should sub directories be ignored when scanning directories for data files. Defaults to true (ignores subdirectories), consistent with Hive. Note that this setting does not affect reading partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`). |
135
135
| datafusion.execution.listing_table_factory_infer_partitions | true | Should a `ListingTable` created through the `ListingTableFactory` infer table partitions from Hive compliant directories. Defaults to true (partition columns are inferred and will be represented in the table schema). |
136
136
| datafusion.execution.enable_recursive_ctes | true | Should DataFusion support recursive CTEs |
0 commit comments