fix: respect DataFrameWriteOptions::with_single_file_output for paths without extensions by kumarUjjawal · Pull Request #19931 · apache/datafusion

kumarUjjawal · 2026-01-21T17:46:26Z

Which issue does this PR close?

Closes Regression: DataFrameWriteOptions::with_single_file_output produces a directory #13323.

Rationale for this change

When using DataFrameWriteOptions::with_single_file_output(true), the setting was being ignored if the output path didn't have a file extension. For example:

df.write_parquet("/path/to/output", 
    DataFrameWriteOptions::new().with_single_file_output(true), 
    None).await?;

Would create a directory /path/to/output/ with files inside instead of a single file at /path/to/output.

This happened because the demuxer used a heuristic based solely on file extension, ignoring the explicit user setting.

What changes are included in this PR?

New FileOutputMode enum: Uses explicit modes (Automatic, SingleFile, Directory) in FileSinkConfig for clearer output path handling.
The demuxer now uses the user's explicit setting instead of always relying on extension-based heuristics.

Are these changes tested?

New unit test test_single_file_output_without_extension tests the fixed behavior
All sqllogictest pass

Are there any user-facing changes?

Breaking for direct FileSinkConfig construction: The struct now requires file_output_mode: FileOutputMode field. Use FileOutputMode::Automatic to preserve existing behavior.

… without extensions

martin-g · 2026-01-22T12:45:09Z

datafusion/proto/src/physical_plan/from_proto.rs

            insert_op,
            keep_partition_by_columns: conf.keep_partition_by_columns,
            file_extension: conf.file_extension.clone(),
+            // For deserialized plans, use extension heuristic (backward compatible)


For how long this backward compatible approach will be used ? At some point it should start using the new config property.

IMO the proper way would be to add a new optional field to the Protobuf format, then try to read it and use None only if it is missing, but if it is not missing then use its value.

Thanks for the feedback. You were right, I made some changes based on the input. Let me know if this looks better.

alamb

Thank you @kumarUjjawal -- this looks good to me

I had a few minor suggestions -- let me know what you think

alamb · 2026-01-23T18:34:38Z

datafusion-examples/examples/data_io/parquet_encrypted.rs

    // Create a temporary file location for the encrypted parquet file
    let tmp_source = TempDir::new()?;
-    let tempfile = tmp_source.path().join("cars_encrypted");
+    let tempfile = tmp_source.path().join("cars_encrypted.parquet");


does it need to be called .parquet even though the dataframe explicitly says write_parquet? Or is this just to clean up the code?

If there is no file extension and there is no configuration (Some(true)) then the heuristics decide that this is a folder and creates a Parquet partition file.
If there is no config and the path has file extension then it writes all the content in this file (no partitions)

Actually why is this change needed for the example? Is it fixing some behaviour made by this change or just clarifying the example?

Before this PR, with_single_file_output(true) was silently ignored for paths without extensions it would create a directory cars_encrypted/ with parquet files inside. The read_parquet() call then found these files in the directory. After this PR, with_single_file_output(true) is correctly respected, creating a single file at the exact path cars_encrypted . However, read_parquet() with default options expects a .parquet extension and fails validation. Adding .parquet to the filename fixes this and makes the example explicit about the expected file format.

alamb · 2026-01-23T18:37:14Z

datafusion/core/src/dataframe/mod.rs

        };

+        // Build copy options, including single_file_output if explicitly set
+        let mut copy_options: HashMap<String, String> = HashMap::new();


since this seems to be a common piece of functionality, what do you think about creating a common function that takes a DataFrameWriteOptions and returns a CopyOptions?

Thank you. I will take a lool at that and see how it shapes.

alamb · 2026-01-23T18:43:03Z

Thank you @martin-g , as always, for the review

alamb · 2026-01-23T19:02:52Z

I took the liberty of adding a small note to the upgrade guide too

Jefffrey · 2026-01-24T05:44:54Z

datafusion/datasource/src/file_sink_config.rs

+    /// Override for single file output behavior.
+    /// - `None`: use extension heuristic (path with extension = single file)
+    /// - `Some(true)`: force single file output at exact path
+    /// - `Some(false)`: force directory output with generated filenames
+    pub single_file_output: Option<bool>,


I think it would be better to make this an enum, something like

enum FileOutputMode { Automatic, SingleFile, Directory, }

It makes it clearer at a glance vs needing to check the comments to know what None, Some(true) and Some(false) mean

Great! Thanks @Jefffrey

Jefffrey

Could you also ensure the PR body is up to date with latest changes? Currently still refers to Option<bool>

Jefffrey · 2026-01-26T04:27:26Z

datafusion-examples/examples/data_io/parquet_encrypted.rs

    // Create a temporary file location for the encrypted parquet file
    let tmp_source = TempDir::new()?;
-    let tempfile = tmp_source.path().join("cars_encrypted");
+    let tempfile = tmp_source.path().join("cars_encrypted.parquet");


Actually why is this change needed for the example? Is it fixing some behaviour made by this change or just clarifying the example?

Jefffrey · 2026-01-26T04:27:47Z

datafusion/core/src/dataframe/mod.rs

        self
    }
+
+    fn copy_to_options(&self) -> HashMap<String, String> {


This is a bit confusingly named for what it does

Jefffrey · 2026-01-26T04:28:09Z

datafusion/core/src/dataframe/parquet.rs

+    /// This verifies the fix for the regression where extension heuristics
+    /// ignored the explicit with_single_file_output(true) setting.
+    #[tokio::test]
+    async fn test_single_file_output_without_extension() -> Result<()> {


Do we have tests for all the modes? automatic, single, directory

Jefffrey · 2026-01-26T04:28:47Z

datafusion/core/src/physical_planner.rs

                };

+                // Parse single_file_output option if explicitly set
+                let file_output_mode = match source_option_tuples


I wonder if it would be better to push this enum into single_file_output itself as well instead of having this parse from an Option<bool> logic 🤔

I kept with_single_file_output(bool) in DataFrameWriteOptions for backward compatibility, it's the existing public API. Internally, I changed the field to Option so we can distinguish "not set" from "explicitly set to false". The conversion to FileOutputMode happens in the physical planner. Adding with_file_output_mode(FileOutputMode) would expand the API surface; are you okay with that?

Jefffrey · 2026-01-26T04:31:34Z

datafusion/core/src/physical_planner.rs

+                // Filter out sink-related options that are not format options
+                let format_options: HashMap<String, String> = source_option_tuples


What's the need for filtering the option here?

The single_file_output option is a sink-level configuration that controls file output behavior, not a format-specific option . The file_type_to_format().create(session_state, &format_options) call validates options against the format's config schema and would fail with "unrecognized option" if we pass single_file_output. So we extract it first for FileSinkConfig, then filter it out before passing the remaining options to the format factory.

Jefffrey · 2026-01-26T04:32:31Z

datafusion/proto/proto/datafusion.proto

+  // When not set, uses extension heuristic (path with extension = single file).
+  // When set to true, forces single file output at exact path.
+  // When set to false, forces directory output with generated filenames.
+  optional bool single_file_output = 12;


Might be better to represent this as an actual enum to more align with the source 🤔

Jefffrey · 2026-01-26T04:33:52Z

datafusion/proto/tests/cases/roundtrip_physical_plan.rs

        insert_op: InsertOp::Overwrite,
        keep_partition_by_columns: true,
        file_extension: "json".into(),
+        file_output_mode: FileOutputMode::Automatic,


Can we use different modes for each of the tests for coverage

Jefffrey · 2026-01-26T04:34:03Z

docs/source/library-user-guide/upgrading.md

+
+### `FileSinkConfig` adds `single_file_output`
+
+`FileSinkConfig` now includes a `single_file_output: Option<bool>` field to override the


Reminder to update this

alamb · 2026-01-27T22:10:18Z

docs/source/library-user-guide/upgrading.md

+*to the main branch and are awaiting release in this version. See [#19692] for
+\*more details.
+
+[#19692]: https://github.com/apache/datafusion/issues/19692


alamb · 2026-01-28T20:57:43Z

Thanks again @kumarUjjawal and @Jefffrey and @martin-g

fix: respect DataFrameWriteOptions::with_single_file_output for paths…

ab87058

… without extensions

github-actions bot added core Core DataFusion crate catalog Related to the catalog crate proto Related to proto crate datasource Changes to the datasource crate labels Jan 21, 2026

fix parquet file load

d4e67d3

martin-g reviewed Jan 22, 2026

View reviewed changes

use proto serialization

e53f51f

martin-g approved these changes Jan 23, 2026

View reviewed changes

alamb approved these changes Jan 23, 2026

View reviewed changes

Add upgrade guide note

bb71255

github-actions bot added the documentation Improvements or additions to documentation label Jan 23, 2026

Jefffrey reviewed Jan 24, 2026

View reviewed changes

kumarUjjawal added 2 commits January 25, 2026 17:16

crete file output mode enum

9731717

fix clippy issue

d30cf77

Jefffrey reviewed Jan 26, 2026

View reviewed changes

kumarUjjawal and others added 4 commits January 26, 2026 12:07

add test and refactor

5824db2

fix docs formatting

e442ad3

Merge branch 'main' into fix/regression_datagrame_single_file

e83fc8e

Merge branch 'main' into fix/regression_datagrame_single_file

ba54292

alamb reviewed Jan 27, 2026

View reviewed changes

alamb added this pull request to the merge queue Jan 28, 2026

Merged via the queue into apache:main with commit 36c0cda Jan 28, 2026
29 checks passed

Jefffrey mentioned this pull request Jan 29, 2026

fix(parquet): write single file if option is set #17009

Closed

		// Filter out sink-related options that are not format options
		let format_options: HashMap<String, String> = source_option_tuples


		### `FileSinkConfig` adds `single_file_output`

		`FileSinkConfig` now includes a `single_file_output: Option<bool>` field to override the

Comments

Conversation

kumarUjjawal commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-g Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 23, 2026

Uh oh!

alamb commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kumarUjjawal commented Jan 21, 2026 •

edited

Loading

martin-g Jan 23, 2026 •

edited

Loading

alamb commented Jan 28, 2026 •

edited

Loading