You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 268fdb9
Browse filesBrowse the repository at this point in the historyBrowse files
fix: address review feedback on the native WriteFilesExec seam
Correctness:
- Decline HDFS writes whose *file names* would diverge, not just their
directory. `mapreduce.output.basename` is caller-controlled and reaches every
committed name through `HadoopMapReduceCommitProtocol.getFilename`; a basename
holding `?` or `#` makes the native URL parser truncate, so every task writes
the same name and they overwrite each other at commit. Adds an execution-time
backstop over the complete `newTaskTempFile` path, which a custom commit
protocol owns and planning cannot predict.
- Read the compression option case-insensitively, as Spark's `ParquetOptions`
does. `option("Compression", "lz4_raw")` used to fall through to the SQLConf
default, so the unsupported-codec guard was bypassed and Comet wrote SNAPPY
into a file Spark had named `.lz4raw.parquet`. The codec is now also
re-derived per task from `CodecConfig.from(taskAttemptContext)`, the same
place the file extension comes from, so the name and the contents agree by
construction. The shared helpers move to `NativeWriteUtils`, which fixes the
identical bug on the Spark 3.x path.
- Use `Utils.tryWithSafeFinallyAndFailureCallbacks` / `tryWithSafeFinally` in
`executeTask` and `writeNatively`, matching `FileFormatWriter`: a failure
while aborting or closing the iterator is attached as a suppressed exception
instead of replacing the failure that caused it. `statsTrackers` moves inside
the guard so a throwing `newTaskInstance` still reaches `abortTask`.
Planning and reporting:
- Convert `WriteFilesExec` from its enclosing `DataWritingCommandExec` rather
than from a separate tag pre-pass, so the output path comes straight from the
command that owns it and neither `withNewChildren` copying tags nor "nothing
hands us a bare WriteFilesExec" has to hold.
- Only skip the fallback reason on `DataWritingCommandExec` when its child
really was converted; a write with no native child now says why it fell back.
- Drop the node's duplicate `files_written`/`bytes_written`/`rows_written`.
`BasicWriteJobStatsTracker` is authoritative here, and the native
`bytes_written` reads 0 on HDFS.
Tests and docs:
- Rust `url_path_rewritten_characters` pins the `url` crate's path encode set,
which the JVM guard mirrors; a crate upgrade can no longer reopen the hole
with a green build.
- New JVM coverage: mixed-case `compression` (honored, and declined when
unsupported), the #3426 nested-name INSERT, an empty non-zero partition
writing no file, the third-party `WriteTaskStatsTracker` warning, and the
basename/committer-path guards.
- The abort test no longer claims to demonstrate task retry or speculation.
- `installation.md` says which Spark versions its EXPLAIN output applies to.
0 commit comments