Skip to content

Commit f1121ca

Browse files
authored
feat: support the S3A magic committer in native Parquet writes (#6940)
* feat: support the S3A magic committer in native Parquet writes On Spark 4.0+, a task file that is neither local nor on HDFS is created by the JVM through Hadoop's FileSystem API, as Spark's Parquet writer creates it, and the native writer streams the file's bytes into that stream over JNI. S3A therefore creates the magic committer's files itself and records their pending uploads. A failed write aborts the stream. Planning admits an S3 destination when the S3A magic committer commits it, through PathOutputCommitProtocol or BindingParquetOutputCommitter. Other S3 writes still fall back to Spark. * test: make the magic committer's failed-write check isolate the writer's abort S3A's job abort also aborts pending uploads under the destination, through core-default.xml's fs.s3a.committer.abort.pending.uploads=true, which hid a writer that never aborted its file. Turn that off in the test, use one task so job abort cannot race other tasks, and run each committer binding alone. Add a unit test that the output stream writer sends nothing after a failed write. * feat: support the S3A magic committer in Spark 3.4 and 3.5 native writes The Spark 3.x writer runs the session's commit protocol with the write's Hadoop configuration since #4746, so it reaches S3A's committers as Spark's writer does. Both writers now create task files through one helper, NativeWriteUtils.writeFile, which creates any file that is neither local nor on HDFS through Hadoop and closes or aborts it, and both serdes share one file system gate. The magic committer tests run on every supported Spark version; in-memory commit tracking is skipped on Hadoop 3.3.4, which lacks it. * refactor: simplify the native write path through the JVM The native writer now fails a task with the output stream's own error, which keeps the Java exception for the JNI boundary to rethrow as it was, so HadoopFileOutput no longer records failures and the JVM no longer swaps exceptions. Both writers write a task file through one helper, CometWriteFilesExec.writeFile, which sets the file and stream flag, runs the plan, and closes or aborts the file. Also: the pusher and output stream registrations share their rules, the writer takes an optional stream, local writes report their counted bytes, planning skips object store options and Hadoop configurations it does not need, the committer factory keys come from Hadoop's constants, and the tests share their setup and read S3A's success manifest through SuccessData. * address review: drop the unreachable bucket committer fallback and say once why S3A creates the file * address review: enrich write errors as Spark does, test the Hadoop route's failure paths Fail a native write's task with TASK_WRITE_FAILED as Spark's writer does, keeping fetch failures and fast-failing on an existing file. Admit subclasses of PathOutputCommitProtocol at planning. Build each task's plan once and create its iterator through CometExec, register the output stream through register_before_execution directly, and move the supported destinations to the Parquet page's Writing section. Test a file of several MiB, a failure partway through it, a failed close, a stream that cannot abort, a retried task and a path with special characters, and against MinIO, a failure after S3A uploaded a part and failed task and job commits. * test: compare what a failed job commit leaves pending with Spark's writer S3A's magic committer leaves the uploads of committed tasks pending when job commit fails, for Spark's own writer as well, so check that the native write leaves the same and nothing visible. Scope each pending upload check to its table, and abort what a test leaves behind. * address review: test the fast-fail and commit paths, nested and temporal columns, and the sink's guards Cover the branches of the Hadoop route that had no test: a task file that already exists, a file that exists at task commit (with and without fastFailOnFileFormatOutput), nested and temporal columns compared with Spark's writer through chunked writes and the abort path, an INSERT into a table of the V2 file source, and the output stream sink's oversize and null checks. * fix: handle the fallible execution context lookup in setOutputStream * test: write complex and temporal types through the Hadoop route * test: leave the non-ASCII character out of the output path when the JVM cannot encode it
1 parent 84944e9 commit f1121ca

31 files changed

Lines changed: 2538 additions & 164 deletions

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -497,6 +497,7 @@ jobs:
497497
- name: "scans"
498498
value: |
499499
org.apache.comet.parquet.CometParquetWriterSuite
500+
org.apache.comet.parquet.CometParquetWriterRetrySuite
500501
org.apache.comet.parquet.CometEmptyRelationParquetWriterSuite
501502
org.apache.spark.sql.comet.CometNativeWriteSuite
502503
org.apache.comet.parquet.ParquetReadV1Suite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ jobs:
163163
- name: "scans"
164164
value: |
165165
org.apache.comet.parquet.CometParquetWriterSuite
166+
org.apache.comet.parquet.CometParquetWriterRetrySuite
166167
org.apache.comet.parquet.CometEmptyRelationParquetWriterSuite
167168
org.apache.spark.sql.comet.CometNativeWriteSuite
168169
org.apache.comet.parquet.ParquetReadV1Suite

‎dev/ci/check-suites.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ def file_to_class_name(path: Path) -> str | None:
3535
ignore_list = [
3636
"org.apache.comet.parquet.ParquetReadSuite", # abstract
3737
"org.apache.comet.parquet.ParquetReadFromS3Suite", # manual test suite
38+
"org.apache.comet.parquet.ParquetWriteWithMagicCommitterSuite", # manual test suite
3839
"org.apache.comet.csv.CsvReadFromS3Suite", # manual test suite
3940
"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite", # manual test suite (loads libhdfs, see #5023)
4041
"org.apache.comet.IcebergReadFromS3Suite", # manual test suite

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -494,7 +494,11 @@ the executor.
494494
in-progress row group (`ArrowWriter::memory_size`). The estimate counts encoded pages, encoder
495495
buffers, dictionaries and Bloom filters. The writer keeps parquet-rs's default row-group limit of
496496
1Mi rows and sets no byte limit, so a wide schema can hold a large row group
497-
([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
497+
([#5304](https://github.com/apache/datafusion-comet/issues/5304)). A file that is neither local
498+
nor on HDFS, such as one the S3A magic committer hands out, is written through a Hadoop output
499+
stream, and the reservation then also includes the 1 MiB buffer that batches the writes to it.
500+
The stream's own buffers, such as S3A's upload buffers (`fs.s3a.fast.upload.buffer`), are not
501+
Comet's and sit outside its pool, as they do for Spark's writer.
498502
- The Iceberg writer keeps one file open per partition in a fanout write, so its reservation grows
499503
with the number of partitions a task writes. iceberg-rust's `ParquetWriter` does not expose
500504
parquet-rs's `memory_size`, only every byte the file has written, its in-progress row group's

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,8 @@ combined with `OR`, which Spark plans as an existence `BroadcastNestedLoopJoinEx
135135

136136
| Operator | Status | Notes |
137137
| ------------------------------------------------------------------------------------------------------------------ | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
138-
| `WriteFilesExec` | ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. |
139-
| `DataWritingCommandExec` | ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. |
138+
| `WriteFilesExec` | ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. See [Parquet](parquet.md#writing). |
139+
| `DataWritingCommandExec` | ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. See [Parquet](parquet.md#writing). |
140140
| Iceberg writes: `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, `ReplaceDataExec` | ⚠️ | Disabled by default (opt-in). `spark.comet.write.iceberg.enabled=true` plans the write as `IcebergWrite` and `IcebergCommit` and writes eligible data files natively (`CometIcebergWrite`). Other writes use Iceberg's Java writer. Covers `INSERT INTO`, `INSERT OVERWRITE`, and copy-on-write `DELETE` / `UPDATE` / `MERGE`. Merge-on-read writes (`WriteDeltaExec`) fall back. See [Iceberg Writes](iceberg.md#writes). |
141141
| `MergeRowsExec` | ⚠️ | Spark 3.5+. Experimental, disabled by default (`spark.comet.exec.mergeRows.enabled`). On Spark 4.1+, stock V2 writers retain Spark MergeRowsExec; Comet's split Iceberg path can run it natively while preserving MergeSummary. See [MERGE INTO](compatibility/operators.md#merge-into-mergerowsexec). |
142142

‎docs/source/user-guide/latest/parquet.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,14 @@ partitioned and bucketed writes fall back to Spark, as do writes with
191191
behavior are not reproduced, as listed under [Known Issues](#known-issues). Iceberg writes use a
192192
different native writer, described on the [Iceberg](iceberg.md#writes) page.
193193

194+
The native writer supports local and HDFS destinations, and S3 writes that the S3A magic committer
195+
commits. Spark 4.1 configures that committer by default when spark-hadoop-cloud is on the
196+
classpath, and earlier versions for a bucket that sets
197+
`fs.s3a.bucket.<bucket>.committer.magic.enabled`. S3A creates each of those files, through the same
198+
call Spark's own writer makes, so the magic committer and S3A's write settings, such as server-side
199+
encryption, ACLs, storage class and object headers, apply exactly as they do for Spark's writer.
200+
Other S3 writes fall back to Spark.
201+
194202
## Tuning
195203

196204
### Filter Pushdown / Late Materialization

‎native/core/src/execution/jni_api.rs‎

Lines changed: 75 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,11 @@
2525
//! `createPlan`, `setShufflePartitionPusher`, `executePlan` and `releasePlan` also depend on
2626
//! upcalls into the JVM (input iterators, the task memory manager, metrics, UDFs and scalar
2727
//! subqueries). Their core functions (`create_execution_context`,
28-
//! `register_shuffle_partition_pusher`, `execute_plan` and `release_plan`) take the JVM objects
29-
//! as JNI global references held by the `ExecutionContext` (`PlanJvmRefs`), and the steps that
30-
//! need an `Env` as callbacks: the memory pool factory, the pusher factory, and `PublishMetrics`.
28+
//! `register_shuffle_partition_pusher`, `execute_plan` and `release_plan`) take the JVM objects as
29+
//! JNI global references held by the `ExecutionContext` (`PlanJvmRefs`), and the steps that need
30+
//! an `Env` as callbacks: the memory pool factory, the pusher factory, and `PublishMetrics`.
31+
//! `setOutputStream` registers a native write's output stream through the same
32+
//! `register_before_execution` check as the pusher.
3133
3234
use super::{serde, utils::SparkArrowConvert};
3335
use crate::{
@@ -39,7 +41,10 @@ use crate::{
3941
serde::to_arrow_datatype,
4042
shuffle::spark_unsafe::row::{process_sorted_partitions, process_sorted_row_partition},
4143
},
42-
jvm_bridge::{JVMClasses, JavaShufflePartitionPusher, ShufflePartitionPusher},
44+
jvm_bridge::{
45+
JVMClasses, JavaOutputStreamSink, JavaShufflePartitionPusher, OutputStreamSink,
46+
ShufflePartitionPusher,
47+
},
4348
};
4449
use std::collections::HashSet;
4550

@@ -586,6 +591,10 @@ struct ExecutionContext {
586591
/// Task-owned remote shuffle callback, registered before native planning starts.
587592
/// The callback owns a JNI global reference and can safely run on Tokio workers.
588593
pub shuffle_partition_pusher: Option<Arc<dyn ShufflePartitionPusher>>,
594+
/// The output stream of the file this plan's Parquet writer writes, registered before native
595+
/// planning starts when the JVM created that file through Hadoop's FileSystem API. Like the
596+
/// shuffle callback, it owns a JNI global reference and can run on any native thread.
597+
pub output_stream: Option<Arc<dyn OutputStreamSink>>,
589598
/// Removes this context's tracing memory-pool entry on every exit path.
590599
memory_pool_registration: Option<ThreadMemoryPoolRegistration>,
591600
}
@@ -898,6 +907,7 @@ fn create_execution_context(
898907
task_context: jvm.task_context,
899908
class_loader: jvm.class_loader,
900909
shuffle_partition_pusher: None,
910+
output_stream: None,
901911
memory_pool_registration,
902912
}))
903913
}
@@ -936,23 +946,72 @@ fn register_shuffle_partition_pusher(
936946
exec_context: &mut ExecutionContext,
937947
make_pusher: impl FnOnce() -> CometResult<Arc<dyn ShufflePartitionPusher>>,
938948
) -> CometResult<()> {
939-
if exec_context.root_op.is_some() {
940-
return Err(CometError::Internal(
941-
"Remote shuffle callback cannot be registered after native execution starts"
942-
.to_string(),
943-
));
949+
register_before_execution(
950+
exec_context.root_op.is_some(),
951+
&mut exec_context.shuffle_partition_pusher,
952+
"Remote shuffle callback",
953+
make_pusher,
954+
)
955+
}
956+
957+
/// Registers one of a plan's task-owned JVM callbacks in `slot`, built by `make` only once the
958+
/// plan is known to accept it: before execution has `started`, and once. `what` names the callback
959+
/// in the errors.
960+
fn register_before_execution<T: ?Sized>(
961+
started: bool,
962+
slot: &mut Option<Arc<T>>,
963+
what: &str,
964+
make: impl FnOnce() -> CometResult<Arc<T>>,
965+
) -> CometResult<()> {
966+
if started {
967+
return Err(CometError::Internal(format!(
968+
"{what} cannot be registered after native execution starts"
969+
)));
944970
}
945971

946-
if exec_context.shuffle_partition_pusher.is_some() {
947-
return Err(CometError::Internal(
948-
"Remote shuffle callback has already been registered for this task".to_string(),
949-
));
972+
if slot.is_some() {
973+
return Err(CometError::Internal(format!(
974+
"{what} has already been registered for this task"
975+
)));
950976
}
951977

952-
exec_context.shuffle_partition_pusher = Some(make_pusher()?);
978+
*slot = Some(make()?);
953979
Ok(())
954980
}
955981

982+
/// Binds the output stream of the file the plan's Parquet writer writes, for a file the JVM
983+
/// created itself, before native execution is initialized.
984+
///
985+
/// Separate from `createPlan` for the same reason as `setShufflePartitionPusher`: only native
986+
/// writes the JVM creates the file for need it, so every other caller keeps the existing ABI.
987+
#[no_mangle]
988+
pub extern "system" fn Java_org_apache_comet_Native_setOutputStream(
989+
e: EnvUnowned,
990+
_class: JClass,
991+
exec_context: jlong,
992+
output: JObject,
993+
) {
994+
try_unwrap_or_throw(&e, |env| {
995+
if exec_context == 0 {
996+
return Err(CometError::NullPointer(
997+
"A native write's output stream requires a valid native execution plan".to_string(),
998+
));
999+
}
1000+
1001+
let exec_context = get_execution_context(exec_context)?;
1002+
register_before_execution(
1003+
exec_context.root_op.is_some(),
1004+
&mut exec_context.output_stream,
1005+
"A native write's output stream",
1006+
|| {
1007+
let sink: Arc<dyn OutputStreamSink> =
1008+
Arc::new(JavaOutputStreamSink::try_new(env, &output)?);
1009+
Ok(sink)
1010+
},
1011+
)
1012+
})
1013+
}
1014+
9561015
/// Skipping is opt-in (`spark.comet.exec.aggregate.skipPartial.enabled`): once DataFusion's probe
9571016
/// decides to skip, it never aggregates again, so a task whose keys repeat after a mostly distinct
9581017
/// start shuffles every later row (#6466).
@@ -1483,7 +1542,8 @@ fn execute_plan(
14831542
.with_sql_text_pool(&exec_context.spark_plan)
14841543
.with_task_context(exec_context.task_context.clone())
14851544
.with_class_loader(exec_context.class_loader.clone())
1486-
.with_shuffle_partition_pusher(exec_context.shuffle_partition_pusher.clone());
1545+
.with_shuffle_partition_pusher(exec_context.shuffle_partition_pusher.clone())
1546+
.with_output_stream(exec_context.output_stream.clone());
14871547
let (scans, shuffle_scans, root_op) = planner.create_plan(
14881548
&exec_context.spark_plan,
14891549
&mut exec_context.input_sources.clone(),

0 commit comments

Comments
 (0)