Repository navigation
Commit 5442c93
authored
feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ (apache#5763)
* feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+
Native writes replace the whole DataWritingCommandExec, which means
InsertIntoHadoopFsRelationCommand.run never runs. Everything that method does
has to be re-implemented inside CometNativeWriteExec: a hardcoded
SQLHadoopMapReduceCommitProtocol (so spark.sql.sources.commitProtocolClass is
ignored), dynamicPartitionOverwrite pinned to false, a hand-ported copy of the
SaveMode logic, a bespoke commit-message accumulator, and its own commitJob
call.
Most of the open native-writer issues are symptoms of that one decision rather
than independent defects. On Spark 4.0+, V1WritesUtils.getWriteFilesOpt matches
the WriteFilesExecBase trait (introduced in 4.0 precisely for this), so a Comet
node that extends it gets driven through FileFormatWriter.executeWrite ->
SparkPlan.executeWrite -> doExecuteWrite, and Spark keeps ownership of
everything above the per-task write.
Spark 3.x has no such trait: getWriteFilesOpt matches the concrete
WriteFilesExec case class, a Comet node there would not be found, and Spark
would silently take FileFormatWriter's non-planned, row-based branch. So the
new seam is additive. CometDataWritingCommand and CometNativeWriteExec are kept
unchanged and remain the 3.4/3.5 path; CometExecRule picks the path by version
and the two never both fire. The legacy path goes away with Spark 3.x support.
Add:
- CometWriteFilesExec, overriding doExecuteWrite and mirroring
FileFormatWriter.executeTask for the parts Comet must do itself: build the
TaskAttemptContext, ask the commit protocol for a path, run the native
writer, drive the stats trackers, commit or abort. Plus the CometWriteFiles
serde and a two-line ShimCometWriteFilesExec in spark-4.x / spark-3.x.
- File paths come from FileCommitProtocol.newTaskTempFile and are used
verbatim, so names match Spark's part-<id>-<uuid>-c000.<codec>.parquet and
committers that track individual files (S3A magic, streaming manifest) work.
- Column names, nullability and field IDs come from
WriteJobDescription.dataColumns rather than the query output, so
INSERT INTO t SELECT a+1 writes the target column's name.
- Byte and row counts come from BasicWriteTaskStatsTracker, which stats files
through the FileSystem API and is therefore correct on HDFS.
- ParquetWriter proto: work_dir is now optional. When set (3.x) the native
writer derives the file name as before; when unset (4.0+) output_path is the
exact file to write and is used verbatim.
- On 4.0+ the opt-in moves to spark.comet.operator.WriteFilesExec
.allowIncompatible, with the old DataWritingCommandExec key kept as a
deprecated alternative. isOperatorAllowIncompat now resolves alternatives,
which the planner's by-name lookup previously bypassed.
AQE re-plans the write command's child and re-inserts a WriteFilesExec above the
node Comet already converted; leaving DataWritingCommandExec in place means
Comet no longer has to guard against the resulting nested native writes.
* refactor: take a single deprecated alternative for operator incompat configs
Matches the reviewed form on apache#5293: ConfigBuilder mutates in place, so the Seq
destructuring was rebinding the same object. Only one operator has an
alternative and there is no reason to expect more.
* fix: escape the output path before parsing it as a URI
The output path reaches both write serdes as `Path.toString`, which decodes
percent escapes: a directory containing a space or a literal `%` yields a
string that is not a valid URI, and `URI.create` throws on it.
On Spark 4.0+ that exception escaped `CometWriteFiles.convert` and failed the
query. On 3.x `CometDataWritingCommand.convert` caught it and silently handed
the write back to Spark, so the native writer was never used for those paths.
Round-trip through `Path` instead, which re-escapes. Only the scheme and
authority reach `extractObjectStoreOptions`, but parsing has to succeed to get
at them.
Also assert that the INSERT INTO visibility test's write actually went native,
rather than inferring it from the read-back.
* fix: decline HDFS writes whose output path needs URI escaping
* fix: also decline Unicode HDFS write destinations
The raw/decoded URI comparison only catches what java.net.URI had to
escape, and java.net.URI leaves non-ASCII path characters alone, so an
hdfs://ns/cafe<U+0301>/output destination was admitted. percent_encoding's
should_percent_encode is !byte.is_ascii() || set.contains(byte), so the
native parser escapes every non-ASCII byte regardless of the encode set
and the writer creates caf%C3%A9 outside Spark's staging directory.
The guard now also declines any character the native parser rewrites. The
ASCII half of that set was determined against the locked url 2.5 crate by
parsing hdfs://ns/pre<c>post/output for every printable ASCII c: space, ",
#, <, >, ?, backtick, { and } are rewritten and the rest survive, so
partition directories and Spark's _temporary attempt paths still qualify.
The comment no longer claims the Java comparison detects the divergence on
its own; both conditions are kept because the Java one still catches a
literal % that the native parser leaves alone.
Tests add accented (precomposed and combining), CJK, emoji and nested
non-ASCII cases plus the remaining escaped ASCII characters, built from
code points since scalastyle forbids non-ASCII source. Disabling the new
condition makes the accented case fail, so the Java comparison alone
demonstrably does not cover it.
* fix: address review feedback on the native WriteFilesExec seam
Correctness:
- Decline HDFS writes whose *file names* would diverge, not just their
directory. `mapreduce.output.basename` is caller-controlled and reaches every
committed name through `HadoopMapReduceCommitProtocol.getFilename`; a basename
holding `?` or `#` makes the native URL parser truncate, so every task writes
the same name and they overwrite each other at commit. Adds an execution-time
backstop over the complete `newTaskTempFile` path, which a custom commit
protocol owns and planning cannot predict.
- Read the compression option case-insensitively, as Spark's `ParquetOptions`
does. `option("Compression", "lz4_raw")` used to fall through to the SQLConf
default, so the unsupported-codec guard was bypassed and Comet wrote SNAPPY
into a file Spark had named `.lz4raw.parquet`. The codec is now also
re-derived per task from `CodecConfig.from(taskAttemptContext)`, the same
place the file extension comes from, so the name and the contents agree by
construction. The shared helpers move to `NativeWriteUtils`, which fixes the
identical bug on the Spark 3.x path.
- Use `Utils.tryWithSafeFinallyAndFailureCallbacks` / `tryWithSafeFinally` in
`executeTask` and `writeNatively`, matching `FileFormatWriter`: a failure
while aborting or closing the iterator is attached as a suppressed exception
instead of replacing the failure that caused it. `statsTrackers` moves inside
the guard so a throwing `newTaskInstance` still reaches `abortTask`.
Planning and reporting:
- Convert `WriteFilesExec` from its enclosing `DataWritingCommandExec` rather
than from a separate tag pre-pass, so the output path comes straight from the
command that owns it and neither `withNewChildren` copying tags nor "nothing
hands us a bare WriteFilesExec" has to hold.
- Only skip the fallback reason on `DataWritingCommandExec` when its child
really was converted; a write with no native child now says why it fell back.
- Drop the node's duplicate `files_written`/`bytes_written`/`rows_written`.
`BasicWriteJobStatsTracker` is authoritative here, and the native
`bytes_written` reads 0 on HDFS.
Tests and docs:
- Rust `url_path_rewritten_characters` pins the `url` crate's path encode set,
which the JVM guard mirrors; a crate upgrade can no longer reopen the hole
with a green build.
- New JVM coverage: mixed-case `compression` (honored, and declined when
unsupported), the apache#3426 nested-name INSERT, an empty non-zero partition
writing no file, the third-party `WriteTaskStatsTracker` warning, and the
basename/committer-path guards.
- The abort test no longer claims to demonstrate task retry or speculation.
- `installation.md` says which Spark versions its EXPLAIN output applies to.
* fix: drop a redundant string interpolator flagged by scalafix RedundantSyntax
* fix: decline percent-bearing HDFS basenames at planning
mapreduce.output.basename=part%foo passed the planning guard and then
failed checkNativeWriteDestination at execution, aborting the job instead
of falling back to Spark's writer. needsNativeUrlEscaping deliberately
excludes % because the url crate leaves it alone, so only Java's
raw-vs-decoded comparison sees it.
The basename now goes through the same hdfsPathDivergence predicate the
task guard uses, applied to the path the basename produces. That closes
the gap for % and for the other characters java.net.URI escapes but the
url crate keeps ([, ^, |), and makes the two guards agree by
construction rather than by two character sets staying in sync.
Also updates CometEmptyRelationParquetWriterSuite, which main added while
this branch was open: a native empty relation is the zero-partition input
CometWriteFilesExec swaps a single-partition RDD in for, so on 4.0+ that
write is accelerated rather than declined.
* fix: drop imports left unused by moving the write assertions into the base1 parent b7f35b6 commit 5442c93
19 files changed
Lines changed: 1845 additions & 129 deletions
File tree
- benchmarks/pyspark
- docs/source/user-guide/latest
- native
- core/src
- execution
- operators
- parquet
- proto/src/proto
- spark/src
- main
- scala/org/apache
- comet
- rules
- serde/operator
- spark/sql/comet
- spark-3.x/org/apache/comet/shims
- spark-4.x/org/apache/comet/shims
- test
- scala/org/apache
- comet/parquet
- spark/sql/comet
- spark-4.x/org/apache/comet/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| 81 | + | |
81 | 82 | | |
82 | 83 | | |
83 | 84 | | |
| |||
106 | 107 | | |
107 | 108 | | |
108 | 109 | | |
| 110 | + | |
109 | 111 | | |
110 | 112 | | |
111 | 113 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
181 | 181 | | |
182 | 182 | | |
183 | 183 | | |
184 | | - | |
| 184 | + | |
185 | 185 | | |
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
189 | | - | |
190 | | - | |
| 189 | + | |
| 190 | + | |
191 | 191 | | |
192 | 192 | | |
193 | 193 | | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
194 | 198 | | |
195 | 199 | | |
196 | 200 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | | - | |
121 | | - | |
122 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
123 | 124 | | |
124 | 125 | | |
125 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
223 | | - | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
224 | 228 | | |
225 | | - | |
226 | | - | |
| 229 | + | |
| 230 | + | |
227 | 231 | | |
228 | 232 | | |
229 | 233 | | |
| |||
250 | 254 | | |
251 | 255 | | |
252 | 256 | | |
253 | | - | |
| 257 | + | |
254 | 258 | | |
255 | 259 | | |
256 | 260 | | |
| |||
510 | 514 | | |
511 | 515 | | |
512 | 516 | | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
519 | | - | |
520 | | - | |
521 | | - | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
522 | 529 | | |
523 | 530 | | |
524 | 531 | | |
| |||
647 | 654 | | |
648 | 655 | | |
649 | 656 | | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
650 | 707 | | |
651 | 708 | | |
652 | 709 | | |
| |||
689 | 746 | | |
690 | 747 | | |
691 | 748 | | |
692 | | - | |
| 749 | + | |
693 | 750 | | |
694 | 751 | | |
695 | 752 | | |
| |||
758 | 815 | | |
759 | 816 | | |
760 | 817 | | |
761 | | - | |
| 818 | + | |
762 | 819 | | |
763 | 820 | | |
764 | 821 | | |
| |||
1018 | 1075 | | |
1019 | 1076 | | |
1020 | 1077 | | |
1021 | | - | |
| 1078 | + | |
1022 | 1079 | | |
1023 | 1080 | | |
1024 | 1081 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1991 | 1991 | | |
1992 | 1992 | | |
1993 | 1993 | | |
1994 | | - | |
1995 | | - | |
1996 | | - | |
1997 | | - | |
1998 | | - | |
| 1994 | + | |
1999 | 1995 | | |
2000 | 1996 | | |
2001 | 1997 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1277 | 1277 | | |
1278 | 1278 | | |
1279 | 1279 | | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
1280 | 1327 | | |
1281 | 1328 | | |
1282 | 1329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
897 | 897 | | |
898 | 898 | | |
899 | 899 | | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
900 | 911 | | |
901 | 912 | | |
902 | 913 | | |
903 | | - | |
904 | | - | |
| 914 | + | |
| 915 | + | |
905 | 916 | | |
906 | 917 | | |
907 | 918 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1024 | 1024 | | |
1025 | 1025 | | |
1026 | 1026 | | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
1027 | 1037 | | |
1028 | 1038 | | |
1029 | 1039 | | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
1030 | 1050 | | |
1031 | 1051 | | |
1032 | 1052 | | |
| |||
1050 | 1070 | | |
1051 | 1071 | | |
1052 | 1072 | | |
1053 | | - | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
1054 | 1076 | | |
1055 | 1077 | | |
1056 | | - | |
| 1078 | + | |
1057 | 1079 | | |
1058 | 1080 | | |
1059 | 1081 | | |
1060 | | - | |
1061 | | - | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
1062 | 1088 | | |
1063 | 1089 | | |
1064 | 1090 | | |
| |||
1082 | 1108 | | |
1083 | 1109 | | |
1084 | 1110 | | |
1085 | | - | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
1086 | 1116 | | |
1087 | 1117 | | |
1088 | 1118 | | |
| |||
0 commit comments