Skip to content

Commit 5442c93

Browse files
authored
feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ (apache#5763)
* feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ Native writes replace the whole DataWritingCommandExec, which means InsertIntoHadoopFsRelationCommand.run never runs. Everything that method does has to be re-implemented inside CometNativeWriteExec: a hardcoded SQLHadoopMapReduceCommitProtocol (so spark.sql.sources.commitProtocolClass is ignored), dynamicPartitionOverwrite pinned to false, a hand-ported copy of the SaveMode logic, a bespoke commit-message accumulator, and its own commitJob call. Most of the open native-writer issues are symptoms of that one decision rather than independent defects. On Spark 4.0+, V1WritesUtils.getWriteFilesOpt matches the WriteFilesExecBase trait (introduced in 4.0 precisely for this), so a Comet node that extends it gets driven through FileFormatWriter.executeWrite -> SparkPlan.executeWrite -> doExecuteWrite, and Spark keeps ownership of everything above the per-task write. Spark 3.x has no such trait: getWriteFilesOpt matches the concrete WriteFilesExec case class, a Comet node there would not be found, and Spark would silently take FileFormatWriter's non-planned, row-based branch. So the new seam is additive. CometDataWritingCommand and CometNativeWriteExec are kept unchanged and remain the 3.4/3.5 path; CometExecRule picks the path by version and the two never both fire. The legacy path goes away with Spark 3.x support. Add: - CometWriteFilesExec, overriding doExecuteWrite and mirroring FileFormatWriter.executeTask for the parts Comet must do itself: build the TaskAttemptContext, ask the commit protocol for a path, run the native writer, drive the stats trackers, commit or abort. Plus the CometWriteFiles serde and a two-line ShimCometWriteFilesExec in spark-4.x / spark-3.x. - File paths come from FileCommitProtocol.newTaskTempFile and are used verbatim, so names match Spark's part-<id>-<uuid>-c000.<codec>.parquet and committers that track individual files (S3A magic, streaming manifest) work. - Column names, nullability and field IDs come from WriteJobDescription.dataColumns rather than the query output, so INSERT INTO t SELECT a+1 writes the target column's name. - Byte and row counts come from BasicWriteTaskStatsTracker, which stats files through the FileSystem API and is therefore correct on HDFS. - ParquetWriter proto: work_dir is now optional. When set (3.x) the native writer derives the file name as before; when unset (4.0+) output_path is the exact file to write and is used verbatim. - On 4.0+ the opt-in moves to spark.comet.operator.WriteFilesExec .allowIncompatible, with the old DataWritingCommandExec key kept as a deprecated alternative. isOperatorAllowIncompat now resolves alternatives, which the planner's by-name lookup previously bypassed. AQE re-plans the write command's child and re-inserts a WriteFilesExec above the node Comet already converted; leaving DataWritingCommandExec in place means Comet no longer has to guard against the resulting nested native writes. * refactor: take a single deprecated alternative for operator incompat configs Matches the reviewed form on apache#5293: ConfigBuilder mutates in place, so the Seq destructuring was rebinding the same object. Only one operator has an alternative and there is no reason to expect more. * fix: escape the output path before parsing it as a URI The output path reaches both write serdes as `Path.toString`, which decodes percent escapes: a directory containing a space or a literal `%` yields a string that is not a valid URI, and `URI.create` throws on it. On Spark 4.0+ that exception escaped `CometWriteFiles.convert` and failed the query. On 3.x `CometDataWritingCommand.convert` caught it and silently handed the write back to Spark, so the native writer was never used for those paths. Round-trip through `Path` instead, which re-escapes. Only the scheme and authority reach `extractObjectStoreOptions`, but parsing has to succeed to get at them. Also assert that the INSERT INTO visibility test's write actually went native, rather than inferring it from the read-back. * fix: decline HDFS writes whose output path needs URI escaping * fix: also decline Unicode HDFS write destinations The raw/decoded URI comparison only catches what java.net.URI had to escape, and java.net.URI leaves non-ASCII path characters alone, so an hdfs://ns/cafe<U+0301>/output destination was admitted. percent_encoding's should_percent_encode is !byte.is_ascii() || set.contains(byte), so the native parser escapes every non-ASCII byte regardless of the encode set and the writer creates caf%C3%A9 outside Spark's staging directory. The guard now also declines any character the native parser rewrites. The ASCII half of that set was determined against the locked url 2.5 crate by parsing hdfs://ns/pre<c>post/output for every printable ASCII c: space, ", #, <, >, ?, backtick, { and } are rewritten and the rest survive, so partition directories and Spark's _temporary attempt paths still qualify. The comment no longer claims the Java comparison detects the divergence on its own; both conditions are kept because the Java one still catches a literal % that the native parser leaves alone. Tests add accented (precomposed and combining), CJK, emoji and nested non-ASCII cases plus the remaining escaped ASCII characters, built from code points since scalastyle forbids non-ASCII source. Disabling the new condition makes the accented case fail, so the Java comparison alone demonstrably does not cover it. * fix: address review feedback on the native WriteFilesExec seam Correctness: - Decline HDFS writes whose *file names* would diverge, not just their directory. `mapreduce.output.basename` is caller-controlled and reaches every committed name through `HadoopMapReduceCommitProtocol.getFilename`; a basename holding `?` or `#` makes the native URL parser truncate, so every task writes the same name and they overwrite each other at commit. Adds an execution-time backstop over the complete `newTaskTempFile` path, which a custom commit protocol owns and planning cannot predict. - Read the compression option case-insensitively, as Spark's `ParquetOptions` does. `option("Compression", "lz4_raw")` used to fall through to the SQLConf default, so the unsupported-codec guard was bypassed and Comet wrote SNAPPY into a file Spark had named `.lz4raw.parquet`. The codec is now also re-derived per task from `CodecConfig.from(taskAttemptContext)`, the same place the file extension comes from, so the name and the contents agree by construction. The shared helpers move to `NativeWriteUtils`, which fixes the identical bug on the Spark 3.x path. - Use `Utils.tryWithSafeFinallyAndFailureCallbacks` / `tryWithSafeFinally` in `executeTask` and `writeNatively`, matching `FileFormatWriter`: a failure while aborting or closing the iterator is attached as a suppressed exception instead of replacing the failure that caused it. `statsTrackers` moves inside the guard so a throwing `newTaskInstance` still reaches `abortTask`. Planning and reporting: - Convert `WriteFilesExec` from its enclosing `DataWritingCommandExec` rather than from a separate tag pre-pass, so the output path comes straight from the command that owns it and neither `withNewChildren` copying tags nor "nothing hands us a bare WriteFilesExec" has to hold. - Only skip the fallback reason on `DataWritingCommandExec` when its child really was converted; a write with no native child now says why it fell back. - Drop the node's duplicate `files_written`/`bytes_written`/`rows_written`. `BasicWriteJobStatsTracker` is authoritative here, and the native `bytes_written` reads 0 on HDFS. Tests and docs: - Rust `url_path_rewritten_characters` pins the `url` crate's path encode set, which the JVM guard mirrors; a crate upgrade can no longer reopen the hole with a green build. - New JVM coverage: mixed-case `compression` (honored, and declined when unsupported), the apache#3426 nested-name INSERT, an empty non-zero partition writing no file, the third-party `WriteTaskStatsTracker` warning, and the basename/committer-path guards. - The abort test no longer claims to demonstrate task retry or speculation. - `installation.md` says which Spark versions its EXPLAIN output applies to. * fix: drop a redundant string interpolator flagged by scalafix RedundantSyntax * fix: decline percent-bearing HDFS basenames at planning mapreduce.output.basename=part%foo passed the planning guard and then failed checkNativeWriteDestination at execution, aborting the job instead of falling back to Spark's writer. needsNativeUrlEscaping deliberately excludes % because the url crate leaves it alone, so only Java's raw-vs-decoded comparison sees it. The basename now goes through the same hdfsPathDivergence predicate the task guard uses, applied to the path the basename produces. That closes the gap for % and for the other characters java.net.URI escapes but the url crate keeps ([, ^, |), and makes the two guards agree by construction rather than by two character sets staying in sync. Also updates CometEmptyRelationParquetWriterSuite, which main added while this branch was open: a native empty relation is the zero-partition input CometWriteFilesExec swaps a single-partition RDD in for, so on 4.0+ that write is accelerated rather than declined. * fix: drop imports left unused by moving the write assertions into the base
1 parent b7f35b6 commit 5442c93

19 files changed

Lines changed: 1845 additions & 129 deletions

File tree

‎benchmarks/pyspark/run_all_benchmarks.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ $SPARK_HOME/bin/spark-submit \
7878
--conf spark.memory.offHeap.enabled=true \
7979
--conf spark.memory.offHeap.size=16g \
8080
--conf spark.comet.enabled=true \
81+
--conf spark.comet.operator.WriteFilesExec.allowIncompatible=true \
8182
--conf spark.comet.operator.DataWritingCommandExec.allowIncompatible=true \
8283
--conf spark.comet.parquet.write.enabled=true \
8384
--conf spark.comet.explain.fallback.log.enabled=true \
@@ -106,6 +107,7 @@ $SPARK_HOME/bin/spark-submit \
106107
--conf spark.memory.offHeap.enabled=true \
107108
--conf spark.memory.offHeap.size=16g \
108109
--conf spark.comet.enabled=true \
110+
--conf spark.comet.operator.WriteFilesExec.allowIncompatible=true \
109111
--conf spark.comet.operator.DataWritingCommandExec.allowIncompatible=true \
110112
--conf spark.comet.parquet.write.enabled=true \
111113
--conf spark.comet.explain.fallback.log.enabled=true \

‎docs/source/user-guide/latest/installation.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -181,16 +181,20 @@ Create a test Parquet source
181181
scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
182182
```
183183

184-
Comet will log output similar to:
184+
Comet will log output similar to this on Spark 4.0 and later:
185185

186186
```shell
187187
INFO core/src/lib.rs: Comet native library version $COMET_VERSION initialized
188188
WARN CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explain.fallback.enabled=false to disable this logging):
189-
Execute InsertIntoHadoopFsRelationCommand [COMET: Native support for operator DataWritingCommandExec is disabled. Set spark.comet.parquet.write.enabled=true to enable it.]
190-
+- WriteFiles
189+
Execute InsertIntoHadoopFsRelationCommand
190+
+- WriteFiles [COMET: Native support for operator WriteFilesExec is disabled. Set spark.comet.parquet.write.enabled=true to enable it.]
191191
+- LocalTableScan [COMET: Native support for operator LocalTableScanExec is disabled. Set spark.comet.exec.localTableScan.enabled=true to enable it.]
192192
```
193193

194+
On Spark 3.4 and 3.5 the native writer replaces the whole write command rather than just the
195+
per-task write, so the same message appears on `Execute InsertIntoHadoopFsRelationCommand` and
196+
names `DataWritingCommandExec`.
197+
194198
Query the data from the test source and check:
195199

196200
- INFO message shows the native Comet library has been initialized.

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -117,9 +117,10 @@ omitted from the tables below and may be reconsidered based on demand:
117117

118118
## Writes
119119

120-
| Operator | Status | Notes |
121-
| ------------------------ | ------ | ----------------------------------------------------------------- |
122-
| `DataWritingCommandExec` | ⚠️ | Experimental native Parquet writes, disabled by default (opt-in). |
120+
| Operator | Status | Notes |
121+
| ------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
122+
| `WriteFilesExec` | ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. |
123+
| `DataWritingCommandExec` | ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. |
123124

124125
## Python and UDF
125126

‎native/core/src/execution/operators/parquet_writer.rs‎

Lines changed: 73 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -220,10 +220,14 @@ impl ParquetWriter {
220220
pub struct ParquetWriterExec {
221221
/// Input execution plan
222222
input: Arc<dyn ExecutionPlan>,
223-
/// Output file path (final destination)
223+
/// Where this task writes. When `work_dir` is set (the Spark 3.x `CometNativeWriteExec`
224+
/// path) this is the write's output directory and is unused; the file name is derived from
225+
/// `work_dir`. Otherwise (Spark 4.0+, `CometWriteFilesExec`) it is the exact path of the file
226+
/// to write, chosen by the JVM commit protocol and used verbatim - this operator then never
227+
/// derives file names of its own.
224228
output_path: String,
225-
/// Working directory for temporary files (used by FileCommitProtocol)
226-
work_dir: String,
229+
/// Working directory for temporary files (used by FileCommitProtocol). Spark 3.x only.
230+
work_dir: Option<String>,
227231
/// Job ID for tracking this write operation
228232
job_id: Option<String>,
229233
/// Task attempt ID for this specific task
@@ -250,7 +254,7 @@ impl ParquetWriterExec {
250254
pub fn try_new(
251255
input: Arc<dyn ExecutionPlan>,
252256
output_path: String,
253-
work_dir: String,
257+
work_dir: Option<String>,
254258
job_id: Option<String>,
255259
task_attempt_id: Option<i32>,
256260
compression: ParquetCompression,
@@ -510,15 +514,18 @@ impl ExecutionPlan for ParquetWriterExec {
510514
Arc::new(Schema::new(fields))
511515
});
512516

513-
// Generate part file name for this partition
514-
// If using FileCommitProtocol (work_dir is set), include task_attempt_id in the filename
515-
let part_file = if let Some(attempt_id) = task_attempt_id {
516-
format!(
517-
"{}/part-{:05}-{:05}.parquet",
518-
work_dir, self.partition_id, attempt_id
519-
)
520-
} else {
521-
format!("{}/part-{:05}.parquet", work_dir, self.partition_id)
517+
let part_file = match &work_dir {
518+
// Spark 4.0+ hands over the exact file to write, chosen by the JVM commit protocol.
519+
None => self.output_path.clone(),
520+
// Spark 3.x hands over a working directory instead and expects the writer to name the
521+
// file; that branch goes away with Spark 3.x support.
522+
Some(work_dir) => match task_attempt_id {
523+
Some(attempt_id) => format!(
524+
"{}/part-{:05}-{:05}.parquet",
525+
work_dir, self.partition_id, attempt_id
526+
),
527+
None => format!("{}/part-{:05}.parquet", work_dir, self.partition_id),
528+
},
522529
};
523530

524531
// Configure writer properties
@@ -647,6 +654,56 @@ mod tests {
647654
);
648655
}
649656

657+
/// Spark 4.0+ hands over the exact file to write rather than a working directory. The writer
658+
/// must use that path verbatim - Spark's commit protocol owns naming and staging, and
659+
/// committers that track individual files depend on the name it chose.
660+
#[tokio::test]
661+
async fn test_parquet_writer_uses_output_path_verbatim_without_work_dir() -> Result<()> {
662+
let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, true)]));
663+
let batch = RecordBatch::try_new(
664+
Arc::clone(&schema),
665+
vec![Arc::new(Int32Array::from(vec![1, 2, 3]))],
666+
)?;
667+
668+
let memory_source = MemorySourceConfig::try_new(&[vec![batch]], Arc::clone(&schema), None)?;
669+
let input = Arc::new(DataSourceExec::new(Arc::new(memory_source)));
670+
let temp_dir = tempfile::tempdir()?;
671+
// A name the writer could never have derived itself, matching Spark's convention.
672+
let file_name = "part-00007-11111111-2222-3333-4444-555555555555-c000.parquet";
673+
let output_path = format!("file://{}/{}", temp_dir.path().display(), file_name);
674+
675+
let writer = ParquetWriterExec::try_new(
676+
input,
677+
output_path,
678+
None, // work_dir: Spark 4.0+ path
679+
None,
680+
None,
681+
ParquetCompression::None,
682+
// A non-zero partition id must not leak into the file name.
683+
3,
684+
vec!["id".to_string()],
685+
None,
686+
HashMap::new(),
687+
)?;
688+
689+
let mut stream = writer.execute(0, SessionContext::new().task_ctx())?;
690+
while stream.try_next().await?.is_some() {}
691+
692+
let written = temp_dir.path().join(file_name);
693+
assert!(
694+
written.exists(),
695+
"expected the writer to use the given path verbatim, found: {:?}",
696+
std::fs::read_dir(temp_dir.path())?
697+
.filter_map(|e| e.ok().map(|e| e.file_name()))
698+
.collect::<Vec<_>>()
699+
);
700+
701+
let reader = SerializedFileReader::new(File::open(written)?)?;
702+
assert_eq!(reader.metadata().file_metadata().num_rows(), 3);
703+
704+
Ok(())
705+
}
706+
650707
#[tokio::test]
651708
async fn test_parquet_writer_preserves_catalyst_schema_in_footer() -> Result<()> {
652709
let values = ListArray::from_iter_primitive::<Int32Type, _, _>([
@@ -689,7 +746,7 @@ mod tests {
689746
let writer = ParquetWriterExec::try_new(
690747
input,
691748
work_dir.clone(),
692-
work_dir,
749+
Some(work_dir),
693750
None,
694751
None,
695752
ParquetCompression::None,
@@ -758,7 +815,7 @@ mod tests {
758815
let writer = ParquetWriterExec::try_new(
759816
input,
760817
work_dir.clone(),
761-
work_dir,
818+
Some(work_dir),
762819
None,
763820
None,
764821
ParquetCompression::None,
@@ -1018,7 +1075,7 @@ mod tests {
10181075
let parquet_writer = ParquetWriterExec::try_new(
10191076
memory_exec,
10201077
output_path,
1021-
work_dir,
1078+
Some(work_dir),
10221079
None, // job_id
10231080
Some(123), // task_attempt_id
10241081
ParquetCompression::None,

‎native/core/src/execution/planner.rs‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1991,11 +1991,7 @@ impl PhysicalPlanner {
19911991
let parquet_writer = Arc::new(ParquetWriterExec::try_new(
19921992
Arc::clone(&child.native_plan),
19931993
writer.output_path.clone(),
1994-
writer
1995-
.work_dir
1996-
.as_ref()
1997-
.expect("work_dir is provided")
1998-
.clone(),
1994+
writer.work_dir.clone(),
19991995
writer.job_id.clone(),
20001996
writer.task_attempt_id,
20011997
codec,

‎native/core/src/parquet/parquet_support.rs‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1277,6 +1277,53 @@ mod tests {
12771277
);
12781278
}
12791279

1280+
/// Pins the set of characters the `url` crate rewrites inside a path.
1281+
///
1282+
/// `create_hdfs_object_store` hands `url.path()` to `object_store::path::Path::parse`, so a
1283+
/// character that the parser escapes - or, for `?` and `#`, treats as a delimiter and
1284+
/// truncates at - makes the native writer create a different file from the one Spark's commit
1285+
/// protocol chose and later commits. Job commit still succeeds, so the data simply ends up
1286+
/// somewhere nobody looks.
1287+
///
1288+
/// `NativeWriteUtils.nativeUrlEscapedAscii` on the JVM side keeps a copy of this set and
1289+
/// declines such destinations during planning. Nothing else would notice if a `url` upgrade
1290+
/// moved a character between the two groups and quietly reopened that hole, so assert the set
1291+
/// here rather than in the Scala test, which can only check the copy against itself.
1292+
#[test]
1293+
fn url_path_rewritten_characters() {
1294+
use url::Url;
1295+
1296+
let rewritten: String = (' '..='~')
1297+
.filter(|c| {
1298+
let url = Url::parse(&format!("hdfs://ns/pre{c}post/output")).unwrap();
1299+
url.path() != format!("/pre{c}post/output")
1300+
})
1301+
.collect();
1302+
1303+
assert_eq!(
1304+
rewritten, " \"#<>?`{}",
1305+
"the ASCII characters `url` rewrites inside a path changed; update \
1306+
NativeWriteUtils.nativeUrlEscapedAscii and its tests to match"
1307+
);
1308+
1309+
// Every non-ASCII byte is percent-encoded regardless of the encode set. This is the half
1310+
// a `java.net.URI` raw/decoded comparison cannot see, because Java leaves non-ASCII path
1311+
// characters alone.
1312+
for name in [
1313+
"caf\u{e9}", // precomposed e-acute
1314+
"cafe\u{301}", // "e" plus a combining acute accent
1315+
"\u{65e5}\u{672c}\u{8a9e}", // CJK
1316+
"\u{1f642}", // astral plane
1317+
] {
1318+
let url = Url::parse(&format!("hdfs://ns/{name}/output")).unwrap();
1319+
assert_ne!(
1320+
url.path(),
1321+
format!("/{name}/output"),
1322+
"expected `url` to percent-encode the non-ASCII name {name}"
1323+
);
1324+
}
1325+
}
1326+
12801327
#[cfg(not(feature = "hdfs-opendal"))]
12811328
#[test]
12821329
fn test_prepare_object_store() {

‎native/proto/src/proto/operator.proto‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -897,11 +897,22 @@ message ShuffleWriter {
897897
}
898898

899899
message ParquetWriter {
900+
// Where this task writes. Two shapes, selected by whether `work_dir` is set:
901+
//
902+
// - Spark 4.0+ (CometWriteFilesExec): `work_dir` is unset and `output_path` is the
903+
// fully-qualified path of the Parquet file to write, set per task from
904+
// FileCommitProtocol.newTaskTempFile. Naming and staging are owned by Spark's commit
905+
// protocol so that task-attempt isolation, speculative execution, and committers that track
906+
// individual files (S3A magic, streaming manifest) all behave as they do for Spark's own
907+
// writer. The native writer uses this path verbatim.
908+
// - Spark 3.x (CometNativeWriteExec): `work_dir` is set and the native writer derives the file
909+
// name from it, the partition id and the task attempt id. `output_path` is the write's
910+
// output directory and is unused natively. Goes away with Spark 3.x support.
900911
string output_path = 1;
901912
CompressionCodec compression = 2;
902913
repeated string column_names = 4;
903-
// Working directory for temporary files (used by FileCommitProtocol)
904-
// If not set, files are written directly to output_path
914+
// Working directory for temporary files (used by FileCommitProtocol). Spark 3.x only; see
915+
// output_path above.
905916
optional string work_dir = 5;
906917
// Job ID for tracking this write operation
907918
optional string job_id = 6;

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1024,9 +1024,29 @@ object CometConf extends ShimCometConf {
10241024
.booleanConf
10251025
.createWithEnvVarOrDefault("ENABLE_COMET_STRICT_TESTING", false)
10261026

1027+
/**
1028+
* Deprecated alternatives for `spark.comet.operator.<name>.allowIncompatible`, keyed by
1029+
* operator name. `CometExecRule` resolves these configs by operator name rather than through
1030+
* the registered `ConfigEntry`, so `isOperatorAllowIncompat` has to consult the alternatives
1031+
* itself - otherwise an old key would read `true` from the entry while the planner saw `false`.
1032+
*/
1033+
private val operatorIncompatAlternatives =
1034+
scala.collection.mutable.Map.empty[String, String]
1035+
1036+
/** Spark 3.x only: native writes replace the whole `DataWritingCommandExec` there. */
10271037
val COMET_OPERATOR_DATA_WRITING_COMMAND_ALLOW_INCOMPAT: ConfigEntry[Boolean] =
10281038
createOperatorIncompatConfig("DataWritingCommandExec")
10291039

1040+
/**
1041+
* Spark 4.0+ only: native writes replace just the `WriteFilesExec` child, so the opt-in moved
1042+
* with the operator. The old `DataWritingCommandExec` key keeps working for anyone who had
1043+
* already enabled the experimental writer.
1044+
*/
1045+
val COMET_OPERATOR_WRITE_FILES_ALLOW_INCOMPAT: ConfigEntry[Boolean] =
1046+
createOperatorIncompatConfig(
1047+
"WriteFilesExec",
1048+
Some(getOperatorAllowIncompatConfigKey("DataWritingCommandExec")))
1049+
10301050
/** Create a config to enable a specific operator */
10311051
private def createExecEnabledConfig(
10321052
exec: String,
@@ -1050,15 +1070,21 @@ object CometConf extends ShimCometConf {
10501070
private def configKeyToEnvVar(configKey: String): String =
10511071
configKey.toUpperCase(Locale.ROOT).replace('.', '_')
10521072

1053-
private def createOperatorIncompatConfig(name: String): ConfigEntry[Boolean] = {
1073+
private def createOperatorIncompatConfig(
1074+
name: String,
1075+
alternative: Option[String] = None): ConfigEntry[Boolean] = {
10541076
val configKey = getOperatorAllowIncompatConfigKey(name)
10551077
val envVar = configKeyToEnvVar(configKey)
1056-
conf(configKey)
1078+
val builder = conf(configKey)
10571079
.category(CATEGORY_EXEC)
10581080
.doc(s"Whether to allow incompatibility for operator: $name. " +
10591081
s"False by default. Can be overridden with $envVar env variable")
1060-
.booleanConf
1061-
.createWithEnvVarOrDefault(envVar, false)
1082+
alternative.foreach { alt =>
1083+
operatorIncompatAlternatives.put(name, alt)
1084+
// ConfigBuilder mutates in place, so the result of withAlternative is `builder` itself.
1085+
builder.withAlternative(alt)
1086+
}
1087+
builder.booleanConf.createWithEnvVarOrDefault(envVar, false)
10621088
}
10631089

10641090
def isExprEnabled(name: String, conf: SQLConf = SQLConf.get): Boolean = {
@@ -1082,7 +1108,11 @@ object CometConf extends ShimCometConf {
10821108
}
10831109

10841110
def isOperatorAllowIncompat(name: String, conf: SQLConf = SQLConf.get): Boolean = {
1085-
getBooleanConf(getOperatorAllowIncompatConfigKey(name), defaultValue = false, conf)
1111+
val value = CometConfDeprecations.readWithAlternatives(
1112+
conf,
1113+
getOperatorAllowIncompatConfigKey(name),
1114+
operatorIncompatAlternatives.get(name).toSeq)
1115+
value != null && value.toLowerCase(Locale.ROOT) == "true"
10861116
}
10871117

10881118
def getOperatorAllowIncompatConfigKey(name: String): String = {

0 commit comments

Comments
 (0)