You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f1121ca
Browse filesBrowse the repository at this point in the historyBrowse files
feat: support the S3A magic committer in native Parquet writes (#6940)
* feat: support the S3A magic committer in native Parquet writes
On Spark 4.0+, a task file that is neither local nor on HDFS is created by
the JVM through Hadoop's FileSystem API, as Spark's Parquet writer creates it,
and the native writer streams the file's bytes into that stream over JNI. S3A
therefore creates the magic committer's files itself and records their pending
uploads. A failed write aborts the stream.
Planning admits an S3 destination when the S3A magic committer commits it,
through PathOutputCommitProtocol or BindingParquetOutputCommitter. Other S3
writes still fall back to Spark.
* test: make the magic committer's failed-write check isolate the writer's abort
S3A's job abort also aborts pending uploads under the destination, through
core-default.xml's fs.s3a.committer.abort.pending.uploads=true, which hid a
writer that never aborted its file. Turn that off in the test, use one task so
job abort cannot race other tasks, and run each committer binding alone. Add a
unit test that the output stream writer sends nothing after a failed write.
* feat: support the S3A magic committer in Spark 3.4 and 3.5 native writes
The Spark 3.x writer runs the session's commit protocol with the write's
Hadoop configuration since #4746, so it reaches S3A's committers as Spark's
writer does. Both writers now create task files through one helper,
NativeWriteUtils.writeFile, which creates any file that is neither local nor on
HDFS through Hadoop and closes or aborts it, and both serdes share one file
system gate. The magic committer tests run on every supported Spark version;
in-memory commit tracking is skipped on Hadoop 3.3.4, which lacks it.
* refactor: simplify the native write path through the JVM
The native writer now fails a task with the output stream's own error, which
keeps the Java exception for the JNI boundary to rethrow as it was, so
HadoopFileOutput no longer records failures and the JVM no longer swaps
exceptions. Both writers write a task file through one helper,
CometWriteFilesExec.writeFile, which sets the file and stream flag, runs the
plan, and closes or aborts the file.
Also: the pusher and output stream registrations share their rules, the
writer takes an optional stream, local writes report their counted bytes,
planning skips object store options and Hadoop configurations it does not
need, the committer factory keys come from Hadoop's constants, and the tests
share their setup and read S3A's success manifest through SuccessData.
* address review: drop the unreachable bucket committer fallback and say once why S3A creates the file
* address review: enrich write errors as Spark does, test the Hadoop route's failure paths
Fail a native write's task with TASK_WRITE_FAILED as Spark's writer does,
keeping fetch failures and fast-failing on an existing file. Admit
subclasses of PathOutputCommitProtocol at planning. Build each task's
plan once and create its iterator through CometExec, register the output
stream through register_before_execution directly, and move the
supported destinations to the Parquet page's Writing section.
Test a file of several MiB, a failure partway through it, a failed close,
a stream that cannot abort, a retried task and a path with special
characters, and against MinIO, a failure after S3A uploaded a part and
failed task and job commits.
* test: compare what a failed job commit leaves pending with Spark's writer
S3A's magic committer leaves the uploads of committed tasks pending when
job commit fails, for Spark's own writer as well, so check that the
native write leaves the same and nothing visible. Scope each pending
upload check to its table, and abort what a test leaves behind.
* address review: test the fast-fail and commit paths, nested and temporal columns, and the sink's guards
Cover the branches of the Hadoop route that had no test: a task file that already exists, a file that exists at task commit (with and without fastFailOnFileFormatOutput), nested and temporal columns compared with Spark's writer through chunked writes and the abort path, an INSERT into a table of the V2 file source, and the output stream sink's oversize and null checks.
* fix: handle the fallible execution context lookup in setOutputStream
* test: write complex and temporal types through the Hadoop route
* test: leave the non-ASCII character out of the output path when the JVM cannot encode it
([#5304](https://github.com/apache/datafusion-comet/issues/5304)). A file that is neither local
498
+
nor on HDFS, such as one the S3A magic committer hands out, is written through a Hadoop output
499
+
stream, and the reservation then also includes the 1 MiB buffer that batches the writes to it.
500
+
The stream's own buffers, such as S3A's upload buffers (`fs.s3a.fast.upload.buffer`), are not
501
+
Comet's and sit outside its pool, as they do for Spark's writer.
498
502
- The Iceberg writer keeps one file open per partition in a fanout write, so its reservation grows
499
503
with the number of partitions a task writes. iceberg-rust's `ParquetWriter` does not expose
500
504
parquet-rs's `memory_size`, only every byte the file has written, its in-progress row group's
|`WriteFilesExec`| ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. |
139
-
|`DataWritingCommandExec`| ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. |
138
+
|`WriteFilesExec`| ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. See [Parquet](parquet.md#writing).|
139
+
|`DataWritingCommandExec`| ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. See [Parquet](parquet.md#writing).|
140
140
| Iceberg writes: `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, `ReplaceDataExec`| ⚠️ | Disabled by default (opt-in). `spark.comet.write.iceberg.enabled=true` plans the write as `IcebergWrite` and `IcebergCommit` and writes eligible data files natively (`CometIcebergWrite`). Other writes use Iceberg's Java writer. Covers `INSERT INTO`, `INSERT OVERWRITE`, and copy-on-write `DELETE` / `UPDATE` / `MERGE`. Merge-on-read writes (`WriteDeltaExec`) fall back. See [Iceberg Writes](iceberg.md#writes). |
141
141
|`MergeRowsExec`| ⚠️ | Spark 3.5+. Experimental, disabled by default (`spark.comet.exec.mergeRows.enabled`). On Spark 4.1+, stock V2 writers retain Spark MergeRowsExec; Comet's split Iceberg path can run it natively while preserving MergeSummary. See [MERGE INTO](compatibility/operators.md#merge-into-mergerowsexec). |
0 commit comments