Skip to content

Commit 268fdb9

Browse files
committed
fix: address review feedback on the native WriteFilesExec seam
Correctness: - Decline HDFS writes whose *file names* would diverge, not just their directory. `mapreduce.output.basename` is caller-controlled and reaches every committed name through `HadoopMapReduceCommitProtocol.getFilename`; a basename holding `?` or `#` makes the native URL parser truncate, so every task writes the same name and they overwrite each other at commit. Adds an execution-time backstop over the complete `newTaskTempFile` path, which a custom commit protocol owns and planning cannot predict. - Read the compression option case-insensitively, as Spark's `ParquetOptions` does. `option("Compression", "lz4_raw")` used to fall through to the SQLConf default, so the unsupported-codec guard was bypassed and Comet wrote SNAPPY into a file Spark had named `.lz4raw.parquet`. The codec is now also re-derived per task from `CodecConfig.from(taskAttemptContext)`, the same place the file extension comes from, so the name and the contents agree by construction. The shared helpers move to `NativeWriteUtils`, which fixes the identical bug on the Spark 3.x path. - Use `Utils.tryWithSafeFinallyAndFailureCallbacks` / `tryWithSafeFinally` in `executeTask` and `writeNatively`, matching `FileFormatWriter`: a failure while aborting or closing the iterator is attached as a suppressed exception instead of replacing the failure that caused it. `statsTrackers` moves inside the guard so a throwing `newTaskInstance` still reaches `abortTask`. Planning and reporting: - Convert `WriteFilesExec` from its enclosing `DataWritingCommandExec` rather than from a separate tag pre-pass, so the output path comes straight from the command that owns it and neither `withNewChildren` copying tags nor "nothing hands us a bare WriteFilesExec" has to hold. - Only skip the fallback reason on `DataWritingCommandExec` when its child really was converted; a write with no native child now says why it fell back. - Drop the node's duplicate `files_written`/`bytes_written`/`rows_written`. `BasicWriteJobStatsTracker` is authoritative here, and the native `bytes_written` reads 0 on HDFS. Tests and docs: - Rust `url_path_rewritten_characters` pins the `url` crate's path encode set, which the JVM guard mirrors; a crate upgrade can no longer reopen the hole with a green build. - New JVM coverage: mixed-case `compression` (honored, and declined when unsupported), the #3426 nested-name INSERT, an empty non-zero partition writing no file, the third-party `WriteTaskStatsTracker` warning, and the basename/committer-path guards. - The abort test no longer claims to demonstrate task retry or speculation. - `installation.md` says which Spark versions its EXPLAIN output applies to.
1 parent 39e02de commit 268fdb9

9 files changed

Lines changed: 547 additions & 146 deletions

File tree

‎docs/source/user-guide/latest/installation.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ Create a test Parquet source
137137
scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
138138
```
139139

140-
Comet will log output similar to:
140+
Comet will log output similar to this on Spark 4.0 and later:
141141

142142
```shell
143143
INFO core/src/lib.rs: Comet native library version $COMET_VERSION initialized
@@ -147,6 +147,10 @@ WARN CometExecRule: Comet cannot execute some parts of this plan natively (set s
147147
+- LocalTableScan [COMET: Native support for operator LocalTableScanExec is disabled. Set spark.comet.exec.localTableScan.enabled=true to enable it.]
148148
```
149149

150+
On Spark 3.4 and 3.5 the native writer replaces the whole write command rather than just the
151+
per-task write, so the same message appears on `Execute InsertIntoHadoopFsRelationCommand` and
152+
names `DataWritingCommandExec`.
153+
150154
Query the data from the test source and check:
151155

152156
- INFO message shows the native Comet library has been initialized.

‎native/core/src/execution/operators/parquet_writer.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -514,11 +514,11 @@ impl ExecutionPlan for ParquetWriterExec {
514514
Arc::new(Schema::new(fields))
515515
});
516516

517-
// Spark 4.0+ hands over the exact file to write, chosen by the JVM commit protocol.
518-
// Spark 3.x hands over a working directory instead and expects the writer to name the
519-
// file; that branch goes away with Spark 3.x support.
520517
let part_file = match &work_dir {
518+
// Spark 4.0+ hands over the exact file to write, chosen by the JVM commit protocol.
521519
None => self.output_path.clone(),
520+
// Spark 3.x hands over a working directory instead and expects the writer to name the
521+
// file; that branch goes away with Spark 3.x support.
522522
Some(work_dir) => match task_attempt_id {
523523
Some(attempt_id) => format!(
524524
"{}/part-{:05}-{:05}.parquet",

‎native/core/src/parquet/parquet_support.rs‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -687,6 +687,53 @@ mod tests {
687687
prepare_object_store_with_configs(runtime_env, url, &HashMap::new())
688688
}
689689

690+
/// Pins the set of characters the `url` crate rewrites inside a path.
691+
///
692+
/// `create_hdfs_object_store` hands `url.path()` to `object_store::path::Path::parse`, so a
693+
/// character that the parser escapes - or, for `?` and `#`, treats as a delimiter and
694+
/// truncates at - makes the native writer create a different file from the one Spark's commit
695+
/// protocol chose and later commits. Job commit still succeeds, so the data simply ends up
696+
/// somewhere nobody looks.
697+
///
698+
/// `NativeWriteUtils.nativeUrlEscapedAscii` on the JVM side keeps a copy of this set and
699+
/// declines such destinations during planning. Nothing else would notice if a `url` upgrade
700+
/// moved a character between the two groups and quietly reopened that hole, so assert the set
701+
/// here rather than in the Scala test, which can only check the copy against itself.
702+
#[test]
703+
fn url_path_rewritten_characters() {
704+
use url::Url;
705+
706+
let rewritten: String = (' '..='~')
707+
.filter(|c| {
708+
let url = Url::parse(&format!("hdfs://ns/pre{c}post/output")).unwrap();
709+
url.path() != format!("/pre{c}post/output")
710+
})
711+
.collect();
712+
713+
assert_eq!(
714+
rewritten, " \"#<>?`{}",
715+
"the ASCII characters `url` rewrites inside a path changed; update \
716+
NativeWriteUtils.nativeUrlEscapedAscii and its tests to match"
717+
);
718+
719+
// Every non-ASCII byte is percent-encoded regardless of the encode set. This is the half
720+
// a `java.net.URI` raw/decoded comparison cannot see, because Java leaves non-ASCII path
721+
// characters alone.
722+
for name in [
723+
"caf\u{e9}", // precomposed e-acute
724+
"cafe\u{301}", // "e" plus a combining acute accent
725+
"\u{65e5}\u{672c}\u{8a9e}", // CJK
726+
"\u{1f642}", // astral plane
727+
] {
728+
let url = Url::parse(&format!("hdfs://ns/{name}/output")).unwrap();
729+
assert_ne!(
730+
url.path(),
731+
format!("/{name}/output"),
732+
"expected `url` to percent-encode the non-ASCII name {name}"
733+
);
734+
}
735+
}
736+
690737
#[cfg(not(feature = "hdfs-opendal"))]
691738
#[test]
692739
fn test_prepare_object_store() {

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -108,10 +108,12 @@ object CometExecRule {
108108

109109
/**
110110
* Output path of the write that a `WriteFilesExec` belongs to, copied from the enclosing
111-
* `InsertIntoHadoopFsRelationCommand`. `WriteFilesExec` itself has no output path, so this is
112-
* how [[org.apache.comet.serde.operator.CometWriteFiles]] learns the target filesystem - and,
113-
* by its absence, that a write comes from some other V1 write command. Only used on Spark 4.0+;
114-
* see `CometWriteFilesExec`.
111+
* `InsertIntoHadoopFsRelationCommand`. `WriteFilesExec` itself has no output path, and
112+
* `CometOperatorSerde` only ever sees the operator, so this is how
113+
* [[org.apache.comet.serde.operator.CometWriteFiles]] learns the target filesystem. Set
114+
* immediately before the one `convertToComet` call that reads it, from the command that owns
115+
* the path; the absence of the tag means the write came from somewhere else and must be
116+
* declined. Only used on Spark 4.0+; see `CometWriteFilesExec`.
115117
*/
116118
val WRITE_OUTPUT_PATH: TreeNodeTag[String] = TreeNodeTag[String]("comet.writeOutputPath")
117119

@@ -402,8 +404,16 @@ case class CometExecRule(session: SparkSession)
402404
// therefore Spark's commit protocol, stats trackers and SaveMode handling - in place.
403405
// `V1WritesUtils.getWriteFilesOpt` matches the `WriteFilesExecBase` trait there, which is
404406
// what lets a Comet node stand in for the write node. See CometWriteFilesExec.
405-
case w: WriteFilesExec if isSpark40Plus =>
406-
convertToComet(w, CometWriteFiles).getOrElse(w)
407+
//
408+
// Matched at the command rather than at the write node so that the output path, which
409+
// `WriteFilesExec` does not carry, comes straight from the command that owns it. Converting
410+
// the child from here also means a `WriteFilesExec` without its enclosing command - which
411+
// nothing produces today - is simply left on Spark instead of being converted against a
412+
// stale or missing tag.
413+
case d @ DataWritingCommandExec(cmd: InsertIntoHadoopFsRelationCommand, w: WriteFilesExec)
414+
if isSpark40Plus =>
415+
w.setTagValue(CometExecRule.WRITE_OUTPUT_PATH, cmd.outputPath.toString)
416+
d.withNewChildren(Seq(convertToComet(w, CometWriteFiles).getOrElse(w)))
407417

408418
// Spark 3.x: `getWriteFilesOpt` matches the concrete `WriteFilesExec` case class, so a
409419
// Comet node can never stand in for the write node. Native writes instead replace the whole
@@ -509,17 +519,20 @@ case class CometExecRule(session: SparkSession)
509519
// Some execs should never be replaced. We include
510520
// these cases specially here so we do not add a misleading 'info' message.
511521
op
512-
case _: WriteFilesExec if !isSpark40Plus =>
513-
// On Spark 3.x the write is converted at the DataWritingCommandExec above, which
514-
// unwraps WriteFilesExec inside convertToComet. Tagging it here would produce a
515-
// spurious "WriteFilesExec is not supported" fallback reason (and a warning when
516-
// COMET_EXPLAIN_FALLBACK_LOG_ENABLED=true) even when the write is fully native. On
517-
// 4.0+ the node was offered to CometWriteFiles above and already carries a reason.
522+
case _: WriteFilesExec =>
523+
// The write is converted at the enclosing DataWritingCommandExec above: on Spark 3.x
524+
// by replacing the whole command, on 4.0+ by converting this child from there.
525+
// Tagging it here would produce a spurious "WriteFilesExec is not supported" fallback
526+
// reason (and a warning when COMET_EXPLAIN_FALLBACK_LOG_ENABLED=true) even when the
527+
// write is fully native; where the write really did fall back on 4.0+, the node
528+
// already carries the reason CometWriteFiles gave.
518529
op
519-
case _: DataWritingCommandExec if isSpark40Plus =>
530+
case d: DataWritingCommandExec
531+
if isSpark40Plus && d.child.isInstanceOf[CometWriteFilesExec] =>
520532
// On Spark 4.0+ DataWritingCommandExec is deliberately left in the plan even for a
521533
// fully native write - Comet replaces only its WriteFilesExec child - so tagging it
522-
// would report an accelerated write as a fallback.
534+
// would report an accelerated write as a fallback. A write whose child was not
535+
// converted still falls through to the default case below and gets a reason.
523536
op
524537
case _ =>
525538
// The operator was not converted to a Comet plan and no serde handler claimed it, so
@@ -541,19 +554,6 @@ case class CometExecRule(session: SparkSession)
541554
}
542555
}
543556

544-
// `WriteFilesExec` does not carry the write's output path, but CometWriteFiles needs it to
545-
// decide whether the target filesystem is supported. Record it from the enclosing command
546-
// before the bottom-up walk reaches the write node. The absence of the tag also tells
547-
// CometWriteFiles that the write is not an InsertIntoHadoopFsRelationCommand and must be
548-
// declined. Only the Spark 4.0+ path consults this tag.
549-
if (isSpark40Plus) {
550-
plan.foreach {
551-
case DataWritingCommandExec(cmd: InsertIntoHadoopFsRelationCommand, w: WriteFilesExec) =>
552-
w.setTagValue(CometExecRule.WRITE_OUTPUT_PATH, cmd.outputPath.toString)
553-
case _ =>
554-
}
555-
}
556-
557557
plan.transformUp { case op =>
558558
val converted = convertNode(op)
559559
// Replace SubqueryBroadcastExec with CometSubqueryBroadcastExec in DPP expressions

‎spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala‎

Lines changed: 10 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,8 @@
1919

2020
package org.apache.comet.serde.operator
2121

22-
import java.util.Locale
23-
2422
import scala.jdk.CollectionConverters._
2523

26-
import org.apache.parquet.hadoop.ParquetOutputFormat
2724
import org.apache.spark.SparkException
2825
import org.apache.spark.sql.comet.{CometNativeExec, CometNativeWriteExec}
2926
import org.apache.spark.sql.execution.command.DataWritingCommandExec
@@ -43,9 +40,6 @@ import org.apache.comet.serde.OperatorOuterClass.Operator
4340
*/
4441
object CometDataWritingCommand extends CometOperatorSerde[DataWritingCommandExec] {
4542

46-
private val supportedCompressionCodes =
47-
Set("none", "uncompressed", "snappy", "lz4", "zstd", "gzip")
48-
4943
override def enabledConfig: Option[ConfigEntry[Boolean]] =
5044
Some(CometConf.COMET_NATIVE_PARQUET_WRITE_ENABLED)
5145

@@ -64,7 +58,9 @@ object CometDataWritingCommand extends CometOperatorSerde[DataWritingCommandExec
6458
}
6559

6660
NativeWriteUtils
67-
.escapedHdfsDestination(cmd.outputPath.toString)
61+
// This writer names its own files `part-<partition>-<attempt>.parquet`, so the
62+
// prefix is fixed rather than read from `mapreduce.output.basename`.
63+
.escapedHdfsDestination(cmd.outputPath.toString, "part")
6864
.foreach(reason => return Unsupported(Some(reason)))
6965

7066
if (cmd.bucketSpec.isDefined) {
@@ -75,8 +71,8 @@ object CometDataWritingCommand extends CometOperatorSerde[DataWritingCommandExec
7571
return Unsupported(Some("Partitioned writes are not supported"))
7672
}
7773

78-
val codec = parseCompressionCodec(cmd)
79-
if (!supportedCompressionCodes.contains(codec)) {
74+
val codec = NativeWriteUtils.parseCompressionCodec(cmd.options)
75+
if (!NativeWriteUtils.supportedCompressionCodecs.contains(codec)) {
8076
return Unsupported(Some(s"Unsupported compression codec: $codec"))
8177
}
8278

@@ -106,14 +102,11 @@ object CometDataWritingCommand extends CometOperatorSerde[DataWritingCommandExec
106102

107103
val outputPath = cmd.outputPath.toString
108104

109-
val codec = parseCompressionCodec(cmd) match {
110-
case "snappy" => OperatorOuterClass.CompressionCodec.Snappy
111-
case "lz4" => OperatorOuterClass.CompressionCodec.Lz4
112-
case "zstd" => OperatorOuterClass.CompressionCodec.Zstd
113-
case "gzip" => OperatorOuterClass.CompressionCodec.Gzip
114-
case "none" | "uncompressed" => OperatorOuterClass.CompressionCodec.None
115-
case other =>
116-
withFallbackReason(op, s"Unsupported compression codec: $other")
105+
val plannedCodec = NativeWriteUtils.parseCompressionCodec(cmd.options)
106+
val codec = NativeWriteUtils.protoCompressionCodec(plannedCodec) match {
107+
case Some(codec) => codec
108+
case None =>
109+
withFallbackReason(op, s"Unsupported compression codec: $plannedCodec")
117110
return None
118111
}
119112

@@ -200,18 +193,4 @@ object CometDataWritingCommand extends CometOperatorSerde[DataWritingCommandExec
200193
CometNativeWriteExec(nativeOp, childPlan, outputPath, cmd.mode, committer, jobId)
201194
}
202195

203-
private def parseCompressionCodec(cmd: InsertIntoHadoopFsRelationCommand) = {
204-
// `compression`, `parquet.compression` (i.e., ParquetOutputFormat.COMPRESSION), and
205-
// `spark.sql.parquet.compression.codec` are in order of precedence from highest to
206-
// lowest, matching Spark's own ParquetOptions.compressionCodecClassName.
207-
cmd.options
208-
.get("compression")
209-
.orElse(cmd.options.get(ParquetOutputFormat.COMPRESSION))
210-
.getOrElse(
211-
SQLConf.get.getConfString(
212-
SQLConf.PARQUET_COMPRESSION.key,
213-
SQLConf.PARQUET_COMPRESSION.defaultValueString))
214-
.toLowerCase(Locale.ROOT)
215-
}
216-
217196
}

‎spark/src/main/scala/org/apache/comet/serde/operator/CometWriteFiles.scala‎

Lines changed: 23 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,8 @@
1919

2020
package org.apache.comet.serde.operator
2121

22-
import java.util.Locale
23-
22+
import org.apache.hadoop.conf.Configuration
2423
import org.apache.hadoop.fs.Path
25-
import org.apache.parquet.hadoop.ParquetOutputFormat
2624
import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
2725
import org.apache.spark.sql.comet.{CometNativeExec, CometWriteFilesExec}
2826
import org.apache.spark.sql.execution.datasources.WriteFilesExec
@@ -43,9 +41,6 @@ import org.apache.comet.serde.OperatorOuterClass.Operator
4341
*/
4442
object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
4543

46-
private val supportedCompressionCodecs =
47-
Set("none", "uncompressed", "snappy", "lz4", "zstd", "gzip")
48-
4944
override def enabledConfig: Option[ConfigEntry[Boolean]] =
5045
Some(CometConf.COMET_NATIVE_PARQUET_WRITE_ENABLED)
5146

@@ -83,7 +78,7 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
8378
}
8479

8580
NativeWriteUtils
86-
.escapedHdfsDestination(outputPath)
81+
.escapedHdfsDestination(outputPath, fileNamePrefix(hadoopConf(op)))
8782
.foreach(reason => return Unsupported(Some(reason)))
8883

8984
if (op.bucketSpec.isDefined) {
@@ -102,8 +97,8 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
10297
Some("Writes with spark.sql.files.maxRecordsPerFile set are not supported"))
10398
}
10499

105-
val codec = parseCompressionCodec(op)
106-
if (!supportedCompressionCodecs.contains(codec)) {
100+
val codec = NativeWriteUtils.parseCompressionCodec(op.options)
101+
if (!NativeWriteUtils.supportedCompressionCodecs.contains(codec)) {
107102
return Unsupported(Some(s"Unsupported compression codec: $codec"))
108103
}
109104

@@ -124,14 +119,13 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
124119
return None
125120
}
126121

127-
val codec = parseCompressionCodec(op) match {
128-
case "snappy" => OperatorOuterClass.CompressionCodec.Snappy
129-
case "lz4" => OperatorOuterClass.CompressionCodec.Lz4
130-
case "zstd" => OperatorOuterClass.CompressionCodec.Zstd
131-
case "gzip" => OperatorOuterClass.CompressionCodec.Gzip
132-
case "none" | "uncompressed" => OperatorOuterClass.CompressionCodec.None
133-
case other =>
134-
withFallbackReason(op, s"Unsupported compression codec: $other")
122+
// Planning-time value only, so that a plan can be inspected without a task context.
123+
// CometWriteFilesExec replaces it per task with the codec Parquet names the file after.
124+
val plannedCodec = NativeWriteUtils.parseCompressionCodec(op.options)
125+
val codec = NativeWriteUtils.protoCompressionCodec(plannedCodec) match {
126+
case Some(codec) => codec
127+
case None =>
128+
withFallbackReason(op, s"Unsupported compression codec: $plannedCodec")
135129
return None
136130
}
137131

@@ -144,13 +138,12 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
144138

145139
// getSupportLevel already declined the write if the tag is absent, so this cannot be empty.
146140
outputPathOf(op).foreach { outputPath =>
147-
val hadoopConf = op.session.sessionState.newHadoopConfWithOptions(op.options)
148141
// The tag holds `Path.toString`, which is not a valid URI string: it leaves spaces and
149142
// literal `%` unescaped, so `URI.create` would throw. Round-tripping through `Path` escapes
150143
// them again. Only the scheme and authority matter to `extractObjectStoreOptions`, but
151144
// parsing has to succeed to get at them.
152145
NativeConfig
153-
.extractObjectStoreOptions(hadoopConf, new Path(outputPath).toUri)
146+
.extractObjectStoreOptions(hadoopConf(op), new Path(outputPath).toUri)
154147
.foreach { case (key, value) => writerOpBuilder.putObjectStoreOptions(key, value) }
155148
}
156149

@@ -170,6 +163,17 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
170163
private def outputPathOf(op: WriteFilesExec): Option[String] =
171164
op.getTagValue(CometExecRule.WRITE_OUTPUT_PATH)
172165

166+
private def hadoopConf(op: WriteFilesExec): Configuration =
167+
op.session.sessionState.newHadoopConfWithOptions(op.options)
168+
169+
/**
170+
* The leading component of every file name this write will produce. Spark's
171+
* `HadoopMapReduceCommitProtocol.getFilename` reads it from the task configuration, so a write
172+
* option or a session-level Hadoop setting can replace the usual `part`.
173+
*/
174+
private def fileNamePrefix(hadoopConf: Configuration): String =
175+
hadoopConf.get(NativeWriteUtils.BASE_OUTPUT_NAME, NativeWriteUtils.DEFAULT_BASE_OUTPUT_NAME)
176+
173177
/**
174178
* Whether Spark would roll to a new file every N rows within a task.
175179
*
@@ -188,18 +192,4 @@ object CometWriteFiles extends CometOperatorSerde[WriteFilesExec] {
188192
.getOrElse(SQLConf.get.maxRecordsPerFile)
189193
maxRecordsPerFile > 0
190194
}
191-
192-
private def parseCompressionCodec(op: WriteFilesExec): String = {
193-
// `compression`, `parquet.compression` (i.e., ParquetOutputFormat.COMPRESSION), and
194-
// `spark.sql.parquet.compression.codec` are in order of precedence from highest to
195-
// lowest, matching Spark's own ParquetOptions.compressionCodecClassName.
196-
op.options
197-
.get("compression")
198-
.orElse(op.options.get(ParquetOutputFormat.COMPRESSION))
199-
.getOrElse(
200-
SQLConf.get.getConfString(
201-
SQLConf.PARQUET_COMPRESSION.key,
202-
SQLConf.PARQUET_COMPRESSION.defaultValueString))
203-
.toLowerCase(Locale.ROOT)
204-
}
205195
}

0 commit comments

Comments
 (0)